Migration and transient scale Unreleased: coming in v1.3

OMENative can move an Instance to another node with no dip in serving capacity. It starts a replacement elsewhere and drains the old Instance once the replacement serves. Use a migration to empty a node before maintenance. A plain kubectl drain evicts the pods first, so the Instance stops serving until its pods run again elsewhere. A transient scale changes a component's Instance count without an edit to the InferenceService, until OME or the autoscaler sets it again.

Both act on the component's InferenceReplica, which OME creates to hold its Instances, so both need a component on OMENative. Request an Instance migration and Request a transient scale give the steps.

The examples use the InferenceService qwen3-0-6b in the namespace qwen3-0-6b, from Serve your first model, with its engine switched to OMENative and running two Instances, 0 and 1. The engine's InferenceReplica is qwen3-0-6b-engine.

How a migration works¶

OMENative moves an Instance by surge: it starts a replacement Instance, the surge, before it removes the old one, the source. Moving Instance 0 of qwen3-0-6b-engine off gpu-node-a, with gpu-node-b as a hint, goes like this:

1. OME gives the surge the lowest unused index, 2. The migration record moves to SurgePending, and OME records a MigrationRequestAccepted event. kubectl ome instance list shows Instance 0 as Migrating and Instance 2 as Creating.
2. OME creates the surge's pods from Instance 0's revision, with the same pod layout. They can't schedule on gpu-node-a, and the scheduler prefers gpu-node-b.
3. When the surge is ready and serving, and has stayed ready for the component's lifecycle.minReadySeconds, the record moves to SurgeReady.
4. OME drains Instance 0, with the record in Draining, and then deletes it.
5. The record ends in Completed, with the message migrated to instance=2, and OME records a MigrationCompleted event.

While the move runs, the InferenceReplica's CURRENT column counts the surge, one Instance over DESIRED:

kubectl get inferencereplica qwen3-0-6b-engine -n qwen3-0-6b

NAME                COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
qwen3-0-6b-engine   engine      2         3         2       2           3d

So a migration needs room for one more whole Instance, even when only one pod of a multi-pod Instance is on the node you empty. If no node has room for the surge, the request fails at its deadline.

The surge keeps its index, so after the move the engine's Instances are 1 and 2, and a later scale-up takes index 0.

Migration requests¶

A migration request asks OMENative to move one Instance off its node. It's an annotation on the InferenceService, not on the InferenceReplica, and three kinds of sender write it:

• kubectl ome migration start, which is alpha, writes the request and fills in from_node. For an Instance whose pods span nodes, --from-node names the node to leave.
• Alfred, OME's alpha GPU cluster caretaker, recommends Instance moves when free GPUs fragment or a node turns unhealthy and, when you let it, asks OMENative to carry them out.
• You, or your own tooling. OME treats a hand-written annotation like any other.

The key is ome.io/migration-request-v1- followed by a request UUID, and the value is a JSON object:

{
  "schemaVersion": "v1",
  "component": "engine",
  "instance": 0,
  "from_node": "gpu-node-a",
  "hint_target_nodes": ["gpu-node-b"],
  "reason": "drain gpu-node-a for maintenance",
  "requested_at": "2026-09-28T08:00:00Z",
  "requested_by": "kubectl-ome"
}

OME's webhook rejects a malformed request, or one for a component the InferenceService doesn't have, with a message that starts with migration request.

OME then records the request in the InferenceReplica's status.migrations as Accepted, adds a row to the audit ConfigMap qwen3-0-6b-ome-migration-audit, and removes the annotation. So a missing annotation means OME has taken the request, not that the Instance has moved. The MigrationRequestAccepted event comes later, when the move starts. A request OME can't act on, such as one for a removed component, gets a MigrationRequestRejected event instead of a record.

Migration policy¶

spec.<component>.lifecycle.migrationPolicy.mode sets whether a component's Instances can move:

The runtime's engineConfig, decoderConfig or routerConfig can set the mode too, and the InferenceService's setting wins (see The engine, decoder and router and MigrationPolicy). This is the examples' InferenceService, with migration turned off for its engine:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-0-6b
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 2
    maxReplicas: 4
    lifecycle:
      migrationPolicy:
        mode: Never

Deadline and caps¶

Each request has a deadline: the time OME accepts it, plus the component's lifecycle.instanceReadyTimeout, or OME's cluster-wide instanceReadyTimeout when the component sets none. The clock runs while a request waits. A paused rollout holds every migration until you resume it, and a request whose deadline passes during the pause fails when you do.

A request that misses its deadline fails with a MigrationExpired event. OME removes the surge, if there is one, and the source goes back to Ready, or to Failed if its pods aren't ready.

A known bug: if neither timeout is set, or the controller's lifecycle settings are invalid, every request fails at once with deadline exceeded in phase Accepted: surge never allocated. Set a timeout first, as Set Instance readiness deadlines shows.

Migrations also use these controller settings, in the lifecycle entry of the inferenceservice-config ConfigMap or the chart's ome.controller.lifecycle:

A request that would go over maxMigrationsPerWindow fails, with a RateLimited event and a message such as migration rate cap reached (10/10 in the last 1h0m0s). Without all three audit settings, as on a kustomize install, a request waits in Accepted until you add them or its deadline passes. OME records a MigrationPolicyUnconfigured event, and sets a condition of the same name on the InferenceReplica. The request stays Accepted shows how to add them.

Automatic relocation¶

OMENative also moves Instances on its own. When a new Instance, or an update of a single-pod Instance, runs out of time or gets stuck, OME can mark it Failed and rebuild it on another node, if:

• all the attempt's pods ran on one node, and none failed for a cause in the workload itself, such as an image that can't be pulled;
• the mode is Auto or Surge;
• the Instance has relocations left in autoMigrate.maxAttempts.

The rebuilt Instance also avoids nodes that earlier relocations left. Each relocation adds a Relocated record to status.migrations and records an AutoMigrationTriggered event. After the last one allowed, which also records AutoMigrationCapReached, OME stops relocating the Instance until it's next Ready. What OME does with a failed Instance has the full rules.

When a request waits or fails¶

Once OME has taken a request, the request waits in Accepted while:

• another migration of the same component runs: OME moves one Instance of a component at a time, oldest request first;
• an update's surge Instance is in flight on the component;
• the source Instance has another operation in progress, has an unscheduled pod, or isn't Ready;
• the audit settings aren't set (see Deadline and caps).

A source Instance that's Failed on the revision it runs keeps the request waiting too, but not mute, since no operation in flight ends that wait by itself. OME writes the wait onto the record's message: waiting for source component=engine instance=0, Failed on its running revision qwen3-0-6b-engine-7b4c2e19 (pod qwen3-0-6b-engine-0-default-0 container ome-container stuck (CrashLoopBackOff)), to be Ready again: and then what holds the Instance's repair — the revision's retry block its rebuild waits on, that no repair is open on it, that the kubelet retries the configuration or image the pod needs in place, or where the retry ladder that re-arms the repair stands. When only you can release the hold, the message ends with reset the Instance or publish a corrected revision. A Warning MigrationDeferred event carries the same message, once per cause, and the move goes on by itself once the Instance is Ready again. What OME does with a failed Instance explains the repairs, and The request stays Accepted says what each hold needs. The deadline keeps running while the request waits.

A from_node that no longer matches doesn't fail a request. When the source Instance was rebuilt on another node after the request, OME records a Normal MigrationFromNodeMismatch event naming both nodes, and moves the Instance off the node it occupies; the record's from_node follows. For an Instance whose pods span nodes, with none of them on from_node, the move leaves the leader's node.

OME fails a request when the source names no node to leave, with no live pods at all or a single-pod Instance's leftover pods on two nodes, when maxMigrationsPerWindow is reached or when the deadline passes. A source that waits on you rather than on a repair fails the request at once, too: one that's Failed on a revision it never started, and one whose running revision is held after its failed attempts — even when its pods read Ready between crashes — since the surge would only rebuild the held revision on another node. Both messages name the revision and where its retries stand, and end with correct or release the revision, then request the move again. The record ends in Failed, with a Warning event that says why: see The migration fails.

When a move can't start now, kubectl ome migration start exits 3 with migration precondition conflicts or is stale; inspect migration status and retry explicitly. The usual causes are a paused rollout, an Instance that isn't Ready or already has a request, and the mode Never. An ome.io/rollout-rollback annotation left from a rollback blocks it too. Refusals lists the others.

A request for a component in RawDeployment or MultiNode (deprecated) mode stays on the InferenceService, since those modes have no InferenceReplica to act on it. Remove it yourself:

kubectl annotate inferenceservice qwen3-0-6b -n qwen3-0-6b ome.io/migration-request-v1-3f9c2a1e-7b4d-4c8e-9a6f-2d1b5e8c4a70-

inferenceservice.ome.io/qwen3-0-6b annotated

Watch migrations¶

kubectl ome migration status shows each migration record and its phase. For older moves, kubectl ome migration history also reads the audit ConfigMap, which keeps the last 200 finished migrations until you delete the InferenceService. Step 3 of Request an Instance migration shows both commands, kubectl ome migration explains their columns and MigrationStatus lists a record's fields.

OME records migration events on the InferenceService. This command lists completed migrations:

kubectl get events -n qwen3-0-6b --field-selector reason=MigrationCompleted -o custom-columns=KIND:.involvedObject.kind,NAME:.involvedObject.name,MESSAGE:.message

KIND               NAME         MESSAGE
InferenceService   qwen3-0-6b   OMENative migration uuid=3f9c2a1e-7b4d-4c8e-9a6f-2d1b5e8c4a70 complete: component=engine instance=0 -> instance=2 (revision=qwen3-0-6b-engine-7b4c2e19)

Transient scale¶

A transient scale sets spec.replicas on a component's InferenceReplica through the Kubernetes scale subresource. The InferenceService's status.components.<component>.scaleTargetRef names the InferenceReplica. Use kubectl ome scale, which checks first, or plain kubectl scale, which skips those checks.

The count lasts until whatever owns it writes it again. To change it for good, change minReplicas, maxReplicas or the autoscaler in the InferenceService.

Who owns the count¶

The component's autoscaler class decides what writes the count next. With no autoscaling settings, the class is HPA, at 80% average CPU utilization. Component autoscaling explains the classes.

kubectl ome scale¶

kubectl ome scale needs --override-autoscaler --yes, whatever the autoscaler class. It reports that the API server stored the count, not that the new Instances are up.

The command refuses while a rollout, a migration or other Instance work is in progress on the component, and when the count is outside the component's bounds. It reads the bounds from the InferenceService and its runtime, not from the controller's replica defaults. So when neither sets maxReplicas, it accepts only minReplicas, or 1 when that's unset. Refusals lists every cause.

When the count changes¶

• When spec.replicas goes up, OME creates Instances at the lowest unused indices.
• When it goes down, OME keeps Ready Instances first and lower indices next, and drains and deletes the rest. It keeps an Instance that's migrating, and its surge.
• A count of 0 runs one Instance until OME sets the count back to minReplicas. kubectl ome scale refuses 0.
• A paused rollout holds scale-up, but OME still scales down.

kubectl get inferencereplica shows the new count under DESIRED.

Next steps¶

• Request an Instance migration: move an Instance off a node, step by step.
• Request a transient scale: change a component's Instance count for a while.
• Alfred: how Alfred finds Instances to move.
• Configure pod disruption budgets: limit how many Instances a node drain takes down at once.