Let Alfred migrate Instances Unreleased: coming in v1.3

Turn on Alfred's execute mode, so that it asks OMENative to migrate Instances off fragmented or unhealthy nodes within the limits you set.

In execute mode, Alfred stops only recommending moves and starts making them. For each move it makes, it writes the same migration request that kubectl ome migration start writes, and OMENative moves the Instance as the component's migration policy says. Alfred is alpha.

Alfred can then move any OMENative Instance, in any namespace, that its defragmentation or node health policy picks and its arbiter admits, unless you opt the workload out. These guardrails stay in place:

Alfred's only change to a workload is one request annotation on its InferenceService, and a migration guard that the chart installs rejects any other. Alfred never cordons or drains a node, and never evicts or deletes a pod: see What Alfred changes in your cluster.
Alfred sends one request at a time, and none while any migration request in the cluster is waiting or in progress, yours included.
Just before it sends a request, Alfred checks the move again, and sends it only when a simulation places every replacement pod on a ready node that the plan allowed.
Alfred follows its requests in a journal until OMENative reports their results, and never cancels or replaces one.
Caps, cooldowns, maintenance windows and a failure backoff bound what Alfred moves: see Safety bounds.

Before you begin¶

Alfred installed with the ome-alfred chart in the ome namespace, in recommend-only mode, and its recommendations reviewed: see Run Alfred in recommend-only mode. Keep every value in the values.yaml you installed it with: helm upgrade -f resets any value that the file leaves out to the chart's default.
Kubernetes 1.30 or newer, for the migration guard.
OME v1.3 or newer, with the InferenceReplica controller running and its migration caps set, as the ome-resources chart does by default. The manifests in config/ leave out the caps, and without them the controller holds every request: see The request stays Accepted.
InferenceServices that declare OMENative themselves, not only through their runtime: with spec.deploymentMode: OMENative, a leader or worker block, or the ome.io/deploymentMode annotation on the component. See Opt in to OMENative. Alfred only advises on RawDeployment and MultiNode (deprecated) components: to let it move a MultiNode component, first move it to OMENative.
For an Instance of more than one pod, the OME scheduler, which is alpha and runs only on Kubernetes 1.35. Alfred can simulate such an Instance only when its pods use the OME scheduler: see Opt in OMENative components.
Helm, kubectl, jq and kubectl-ome.
Permission to upgrade the ome-alfred release and edit ConfigMaps in ome, to read ValidatingAdmissionPolicies, their bindings and InferenceServices cluster-wide, and to read InferenceReplicas, events and ConfigMaps in your workloads' namespaces.

The examples use an InferenceService chat in the namespace prod, whose engine is OMENative.

Step 1: Configure the simulation workers¶

Alfred sends a request only when a simulation of your scheduler places every replacement pod, so set up the simulation first, as Set up scheduler simulation for Alfred shows. Once its predictions show in the recommendation record, go on to Step 2 with the same values.yaml.

Step 2: Turn on the migration API¶

migration.apiVersion: v1 tells Alfred that your OME manager accepts v1 migration requests, which Alfred doesn't check. With it, the chart installs the migration guard and the dispatch journal, and Alfred plans each move with target nodes. It still sends nothing while mode is recommend-only.

Add the block to your values.yaml:

values.yaml

migration:
  apiVersion: v1

Upgrade the release:

Shell

helm upgrade ome-alfred oci://ghcr.io/moirai-internal/charts/ome-alfred \
  --version 1.3.0 --namespace ome -f values.yaml

Helm upgrades the release and prints its status. Wait for the new pods:

Shell

kubectl rollout status deployment/ome-alfred -n ome

Output

deployment "ome-alfred" successfully rolled out

When the rollout doesn't finish, see Alfred exits at startup.

From the next decision pass, Alfred records the moves that it would send as advice with the reason SimulationRecommendOnly, and their preferred target nodes as hintTargets:

Shell

kubectl get configmap alfred-recommendations -n ome -o jsonpath='{.data.last-cycle\.json}' \
  | jq '.recommendations[] | select(.advisoryReason == "SimulationRecommendOnly") | {workload, component, instance, reason, fromNode, hintTargets}'

Output

{
  "workload": "prod/chat",
  "component": "engine",
  "instance": 3,
  "reason": "Fragmentation",
  "fromNode": "node-a",
  "hintTargets": [
    "node-c",
    "node-d"
  ]
}

Review these moves: in execute mode, Alfred sends the ones that pass its checks, one at a time. In this release, kubectl ome admin recommendations wrongly shows this record as Partial, so read it with jq.

Check that the guard is in place:

Shell

kubectl get validatingadmissionpolicy,validatingadmissionpolicybinding ome-alfred-migration-writes -o name

Output

validatingadmissionpolicy.admissionregistration.k8s.io/ome-alfred-migration-writes
validatingadmissionpolicybinding.admissionregistration.k8s.io/ome-alfred-migration-writes

Then check the journal:

Shell

kubectl get configmap alfred-dispatch-state -n ome -o jsonpath='{.data.state\.json}'

Output

{"version":"v1","entries":[]}

The chart keeps the journal across upgrades and uninstalls, so a journal from an earlier install can already have entries.

Step 3: Switch Alfred to execute mode¶

Before you switch, decide how much Alfred may move. By default, it starts no migration once 10 have started in the cluster in the last hour, whoever started them: to go slower, lower alfredConfig.maxMigrationsPerHour. To let Alfred move only the InferenceServices you pick, set alfredConfig.defaultMovable to false, and annotate them alfred.ome.io/movable: "true": see Workload annotations. You can also limit defragmentation to maintenance windows, which node-health moves ignore.

Set alfredConfig.mode to execute. With the earlier steps, your values.yaml now reads as follows, next to any values you set before:

values.yaml

global:
  hub: registry.example.com/ome
image:
  tag: v1.3.0@sha256:<digest>
simulation:
  configMapName: alfred-simulation-1
migration:
  apiVersion: v1
alfredConfig:
  mode: execute
  scheduling:
    profiles:
      default-scheduler:
        backend: kube-v135
        schedulerVersion: v1.35.4
        configurationID: sha256:<default-id>
      ome-scheduler:
        backend: ome-v135
        schedulerVersion: v1.35.4
        configurationID: sha256:<ome-id>
        gangScheduling: true

Upgrade the release, and wait for the new pods:

Shell

helm upgrade ome-alfred oci://ghcr.io/moirai-internal/charts/ome-alfred \
  --version 1.3.0 --namespace ome -f values.yaml

Shell

kubectl rollout status deployment/ome-alfred -n ome

Output

deployment "ome-alfred" successfully rolled out

After the next decision pass, the record shows the new mode:

Shell

kubectl get configmap alfred-recommendations -n ome -o jsonpath='{.data.last-cycle\.json}' | jq -r .mode

Output

execute

From now on, each admitted move's outcome is its request's status, such as submitted, or withheld with a dispatchReason. A move that the arbiter turns down is rejected, with a rejectReason: see The arbiter.

Step 4: Watch a migration¶

Alfred sends its first request once a policy picks a move that the arbiter admits and that passes the checks. It records an event on the InferenceService at each step of a request:

Shell

kubectl get events -n prod --field-selector involvedObject.name=chat,source=alfred -o custom-columns=TYPE:.type,REASON:.reason,MESSAGE:.message

Output

TYPE     REASON               MESSAGE
Normal   MigrationSubmitted   defragmentation migration request uuid=7c9e6679-7425-40de-944b-e07fc1f90ae7 for prod/chat/engine instance 3: submitted (RequestSubmitted)

MigrationAcknowledged and MigrationCompleted follow, or the Warnings MigrationFailed or MigrationStalled: see On an InferenceService.

The recommendation record shows each admitted move, with its request's status and ID:

Shell

kubectl get configmap alfred-recommendations -n ome -o jsonpath='{.data.last-cycle\.json}' \
  | jq '.recommendations[] | select(.dispatchStatus) | {workload, component, instance, dispatchStatus, dispatchReason, requestUUID}'

Output

{
  "workload": "prod/chat",
  "component": "engine",
  "instance": 3,
  "dispatchStatus": "submitted",
  "dispatchReason": "RequestSubmitted",
  "requestUUID": "7c9e6679-7425-40de-944b-e07fc1f90ae7"
}

To block until the migration ends, pass its requestUUID to kubectl ome wait:

Shell

kubectl ome wait chat -n prod --for=migration=terminal --request-id=7c9e6679-7425-40de-944b-e07fc1f90ae7 --timeout=30m

The command exits 0 once the migration ends, even when it failed, so check the outcome:

Shell

kubectl ome migration status chat -n prod --component engine

Output

SUBJECT/COMP      STATUS                       DETAIL
7c9e6679/engine   Completed/Terminal/Current   MSG: migrated to instan...

Step 3: Follow the request explains these columns. After a request fails or stalls, Alfred waits migration.failureBackoff, 5 minutes by default, before it sends another.

Troubleshooting¶

The chart fails to render¶

helm upgrade stops with an error that ends in one of these messages:

migration.apiVersion must be v1 when enabled: set migration.apiVersion to v1, or leave it empty.
migration.apiVersion requires simulation.configMapName: set up the simulation first, as in Step 1.
existing alfred-dispatch-state is missing state.json; restore its journal before enabling migration: put the journal's state.json back before you set migration.apiVersion.

Moves stay advisory¶

Check each move's advisoryReason in the record:

OMENativeUnavailable: migration.apiVersion isn't set. See Step 2.
SimulationRecommendOnly: mode is still recommend-only, or the record is older than your upgrade. See Step 3.
A scheduling reason, such as ProfileNotConfigured or GangUnsupported: the simulation has no prediction for the move. See Check the predictions.
RawDeploymentMigrationUnsupported on an OMENative component: the component is OMENative only through its runtime. See Before you begin.
MigrationSurfaceDisabled: alfredConfig.omenativeMigrationEnabled is false.
OMENativeStateIneligible: a migration of the InferenceService ended less than 30 minutes ago, the default perWorkloadCooldownMinutes. Or the InferenceService isn't movable, the component's migration policy is Never, or the component isn't steady, for example during a rollout.

Which Instances Alfred can move gives the conditions.

Admitted moves are withheld¶

Check each withheld move's dispatchReason in the record:

Reason	What to do
`GuardUnavailable`	The guard is missing, changed, or not type-checked yet. Check it as in Step 2, then compare its generations, as below.
`JournalUnavailable`	Alfred can't read or write `alfred-dispatch-state`, or its `state.json` is invalid.
`SnapshotUnavailable`	Alfred can't copy the cluster, for example because the PodGroup CRD is missing.
`SourceUnsupported`	No profile fits the Instance's pods, their node affinity still differs from the InferenceReplica's template after an earlier migration, or the node's `kubernetes.io/hostname` label differs from its name.
`SimulationNotFeasible`, `PredictedTargetUnsafe`	The last simulation didn't place every replacement pod on a ready node that the plan allowed.
`UnresolvedRequest`, `MultipleUnresolvedRequests`	One of Alfred's requests is still under way, as expected during a migration. If it has stalled, see A request is stalled.
`FailureBackoff`	One of Alfred's requests failed or stalled less than `migration.failureBackoff` ago.
`InFlightCap`	A migration request is waiting or in progress somewhere in the cluster. List the waiting requests, as below.
`InvalidCooldown`	An InferenceService has an invalid `alfred.ome.io/cooldown-minutes` annotation. Find it, as below.
`SerialDispatchLimit`	Alfred tried another move in this pass. When the tried move shows it too, its request didn't reach the InferenceService: see A request is stalled.
Another reason	See Dispatch reasons.

When moves show GuardUnavailable while the guard is in place, compare the policy's generations:

Shell

kubectl get validatingadmissionpolicy ome-alfred-migration-writes \
  -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}'

Output

1 1

Alfred uses the guard only when the two numbers are equal, and the policy's status.typeChecking lists no expressionWarnings.

To find the requests behind InFlightCap, list the waiting request annotations:

Shell

kubectl get inferenceservices -A -o json \
  | jq -r '.items[] | .metadata as $m | ($m.annotations // {}) | keys[] | select(startswith("ome.io/migration-request-v1-")) | "\($m.namespace)/\($m.name) \(.)"'

Output

prod/chat ome.io/migration-request-v1-7c9e6679-7425-40de-944b-e07fc1f90ae7

The command prints nothing when no request is waiting. Remove an annotation that nothing acts on, such as one that names a component that isn't OMENative. Migrations already under way show in kubectl ome migration status instead.

To find an invalid cooldown annotation, list them all, with their values:

Shell

kubectl get inferenceservices -A -o json \
  | jq -r '.items[] | select(.metadata.annotations["alfred.ome.io/cooldown-minutes"] != null) | "\(.metadata.namespace)/\(.metadata.name) \(.metadata.annotations["alfred.ome.io/cooldown-minutes"])"'

Output

prod/chat 30m

A valid value is a whole number of minutes, such as 30. Fix or remove any other.

A request is stalled¶

A stalled request stays unresolved, so Alfred withholds every later move with UnresolvedRequest. Alfred resolves it once the InferenceReplica's status reports that the migration completed or failed. Find the stalled entry in the journal:

Shell

kubectl get configmap alfred-dispatch-state -n ome -o jsonpath='{.data.state\.json}' \
  | jq '.entries[] | select(.phase == "stalled") | {uuid, workload, component, instance, reason, workloadUID, irUID}'

Output

{
  "uuid": "9b2d4e71-5c3a-4f8e-a1d2-6e7f8a9b0c1d",
  "workload": {
    "Namespace": "prod",
    "Name": "chat"
  },
  "component": "engine",
  "instance": 3,
  "reason": "AcknowledgementTimeout",
  "workloadUID": "<isvc-uid>",
  "irUID": "<ir-uid>"
}

Then check what OME recorded for the InferenceService's migrations:

Shell

kubectl ome migration history chat -n prod

Output

EVID    REQUEST/COMP   PHASE/STATE   WHEN           DETAIL
AUTH    7c9e6679/E     Completed/T   09-28T10:31Z   migrated to insta...
AUDIT   7c9e6679/E     Completed/T   09-28T10:31Z   migrated
AUDIT   3f2a9c1e/E     Failed/T      09-28T09:20Z   deadline exceeded...

Here, the history has no row for the stalled request, 9b2d4e71. What to do depends on the entry's reason:

Reason	What to do
`ConsumerDeadlineExceeded`	Wait: the migration is still under way after its deadline, and the request resolves when it ends.
`AcknowledgementTimeout`	Look for the request's annotation, as in Admitted moves are withheld. While it's there, the request goes ahead once the InferenceReplica controller runs. With neither the annotation nor a history row, clear the entry.
`OwnerUnavailable`, `ReplicaUnavailable`, `ReplicaIdentityChanged`	Clear the entry once the InferenceService or InferenceReplica is gone, or its `metadata.uid` differs from the entry's `workloadUID` or `irUID`. Check first: a failed read gives the first two reasons too.
Another reason	Clear the entry once the history shows that the request ended.

To clear an entry, edit the journal. In its state.json key, remove the stalled entry from entries, and leave the rest as it is, including "entries": [] when no entry is left:

Shell

kubectl edit configmap alfred-dispatch-state -n ome

Output

configmap/alfred-dispatch-state edited

Alfred reads the journal again at the start of each decision pass. Keep state.json valid: while Alfred can't read it, it withholds every move with JournalUnavailable. Journal states explains the reasons.

Alfred's migrations fail¶

The journal shows the request as failed, and kubectl ome migration status shows OMENative's message. When every migration fails at once with deadline exceeded in phase Accepted: surge never allocated, no instance-ready timeout is set: see Deadline. For other messages, see The migration fails.

Alfred's target nodes are preferences, not reservations: another pod can take one first, and a migration that can't finish by its deadline fails. When too many of Alfred's recent migrations fail, the arbiter rejects every candidate with CircuitBreakerOpen for a while: see Global limits.

Clean up¶

To stop Alfred from sending requests, set alfredConfig.mode back to recommend-only in your values.yaml, and upgrade the release:

Shell

helm upgrade ome-alfred oci://ghcr.io/moirai-internal/charts/ome-alfred \
  --version 1.3.0 --namespace ome -f values.yaml

Shell

kubectl rollout status deployment/ome-alfred -n ome

Output

deployment "ome-alfred" successfully rolled out

Alfred then sends no request, but still follows the requests in its journal.

To turn migration off as well, first wait until this command, which lists the unresolved requests in the journal, prints nothing:

Shell

kubectl get configmap alfred-dispatch-state -n ome -o jsonpath='{.data.state\.json}' \
  | jq -r '.entries[] | select(.phase != "completed" and .phase != "failed") | .uuid'

A stalled request can stay unresolved for good: see A request is stalled. Then remove the migration block from your values.yaml, and upgrade again. The chart removes the guard, and keeps the journal.

Keep the journal, even after you uninstall Alfred: when you turn migration on again, its unresolved requests still hold back new ones. To remove the simulation too, see Set up scheduler simulation for Alfred.

Next steps¶

Alfred policies: the policies, the arbiter and the safety bounds.
Alfred configuration: every chart value and alfredConfig setting.
Alfred metrics and events: what Alfred reports.
Request an instance migration: write a request yourself, and follow it.