kubectl ome migration Unreleased: coming in v1.3

kubectl ome migration shows the migrations of an InferenceService whose components run on OMENative, and its alpha start action asks OME to move an Instance to another node. OME starts a replacement Instance elsewhere, waits until it's ready and serving, then drains and removes the old one, so the component keeps its serving capacity. A multi-node Instance, a leader and its workers, moves as one unit. Migration and transient scale explains how migrations work, and Deployment modes shows how to opt in to OMENative.

kubectl ome migration SUBCOMMAND INFERENCESERVICE [flags]

Each subcommand takes the name of one InferenceService. status and history print a table, or the whole report with -o json or -o yaml, as Output formats describes. Required RBAC lists the permissions each command needs, and start also needs the patch rule in Guarded actions.

Alfred, OME's alpha GPU cluster caretaker, recommends Instance moves when free GPUs fragment or a node turns unhealthy and, when you let it, asks OMENative to carry them out. It sends the same request as start, so status and history show its migrations too. While any migration request waits or runs in the cluster, yours included, Alfred sends none of its own. Global limits lists its other limits.

status¶

kubectl ome migration status INFERENCESERVICE [flags]

status shows the migrations recorded in the InferenceReplicas' status.migrations: those running, and those that ended within the audit window. The ome-resources chart sets that window to one hour, in ome.controller.lifecycle.audit.window. A request shows up once OME accepts it. For older migrations, use history.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has a row for each record's message and one for each issue, with the issue's code in DETAIL:

The report's state is Partial when there's any issue, otherwise Empty when there are no records, and Reported when there are.

The phase says where the migration is. How a migration works describes each step.

The classification is Active while the migration runs, Terminal once it has ended, and Invalid for a malformed record, whose issue codes say what's wrong.

A request that stays in Accepted is waiting: The request stays Accepted explains why, and what to do.

The freshness says whether the InferenceReplica's status has caught up with its spec:

-o json and -o yaml print a MigrationStatusReport:

In a record, the trigger is Manual for a request and Auto for a relocation. Its startedAt is when OME accepted the request, allocatedAt when the replacement got its index, deadline when the migration fails if it hasn't completed, and completedAt when it ended. Deadline and caps explains the deadline. The outcome is InProgress while the migration runs, then Completed, Failed, Relocated, or RelocationConfirmed for a relocation OME marked as succeeded, and Unknown for an invalid record. The report doesn't print the request's reason.

The issue codes are:

Examples¶

Show the migrations of chat in prod, while OME waits for a replacement's pods:

kubectl ome migration status chat -n prod

SUBJECT/COMP      STATUS                        DETAIL
7c9e6679/engine   SurgePending/Active/Current   MSG: surge allocated; w...

The engine's request 7c9e6679 has its replacement, and OME's message is surge allocated; waiting for surge pods. Once the migration completes, the same command shows:

SUBJECT/COMP      STATUS                       DETAIL
7c9e6679/engine   Completed/Terminal/Current   MSG: migrated to instan...

The full message is migrated to instance=4: Instance 4 serves in place of the Instance that moved. For a service with no migration records, the command shows:

SUBJECT/COMP   STATUS          DETAIL
-/-            Empty/Current   -

history¶

kubectl ome migration history INFERENCESERVICE [flags]

history shows the migration records an InferenceService keeps, from three sources, and marks each record with its source:

Only AUTH records are authoritative. An AUDIT entry that's still Started doesn't show on its own that the migration runs.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has a row for each record, and a row for each issue:

The phase is the one the source records: a status phase for AUTH, and Started, Completed or Failed for AUDIT. An AUDIT record's message is its outcome, such as migrated. The report's state is Partial when there's any issue or invalid record, otherwise Reported when there are records, and Empty when there are none.

-o wide prints one row per record, with every field in full. It also adds a row for each issue, and one for each source without records. When that source is unavailable, the row's ISSUES cell gives the reason, such as NotFound or Forbidden. A missing audit ConfigMap, as on a service that never had a migration, shows NotFound. It isn't an issue, and the report can still be Reported.

-o json and -o yaml print a MigrationHistoryReport with the three sources, a summary, the records and the issues.

The issue codes are:

Examples¶

Show the migration history of chat in prod:

kubectl ome migration history chat -n prod

EVID    REQUEST/COMP   PHASE/STATE   WHEN           DETAIL
AUTH    7c9e6679/E     Completed/T   09-28T10:31Z   migrated to insta...
AUDIT   7c9e6679/E     Completed/T   09-28T10:31Z   migrated
AUDIT   3f2a9c1e/E     Failed/T      09-28T09:20Z   deadline exceeded...

The engine's request 7c9e6679 completed at 10:31 UTC. The InferenceReplica still holds its record, and the audit ConfigMap has its entry. The earlier request 3f2a9c1e failed at 09:20, when its deadline passed before the replacement's pods became ready. Its record has left the InferenceReplica's status, so only the audit ConfigMap still has it.

start¶

kubectl ome migration start INFERENCESERVICE --component COMPONENT --instance INDEX [flags]

start asks OME to migrate one Instance of an OMENative component. It adds the request to the InferenceService as an annotation, ome.io/migration-request-v1- followed by a new request ID, and changes nothing else. OME records the request in the InferenceReplica's status, removes the annotation and runs the migration. Migration requests describes the annotation, and Request an Instance migration follows a request from start to end.

The Instance must be Ready on the component's current revision, with no operation in progress. To find its index, run kubectl ome instance list, or list the node's pods as Request an Instance migration shows. The CLI takes the node the Instance's pods run on as the source. When they span several nodes, as a leader and its workers can, name one with --from-node. The replacement prefers the --hint-node nodes, all equally. It can still land elsewhere, but never on the source node. Since it starts before the old Instance drains, the cluster needs room for one more whole Instance off the source node, or the migration fails at its deadline.

start is one of the guarded actions, and follows their contract. To follow the request, run status, or block until it ends with kubectl ome wait and the request ID.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

The values have these limits:

• --instance is a whole number without a sign or leading zeros.
• --from-node is a node name that hosts one of the Instance's pods.
• --hint-node takes up to eight node names, comma-separated or in repeated flags, each once, and not the source node.
• --reason is printable text up to 256 bytes, without :// or anything that looks like a credential or token.
• --requested-by is up to 128 letters, digits and _.:/@+- characters. It's a label, not your identity.
• --request-id is a UUID in canonical form. It looks up a request you already sent, and never sets the ID of a new one.

How start runs¶

start runs the shared steps of a guarded action:

1. The CLI checks the flags, reads what the migration needs, and refuses if the migration isn't safe now, as Refusals lists.
2. It generates the request ID and prints a preview of the request to stderr.
3. It asks Confirm this exact action? [y/N] on stderr, and goes ahead only on y or yes. --yes confirms without the prompt, and scripts need it.
4. It reads the InferenceReplica, the Instance's pods and its ControllerRevision again, and exits 3 if any of them changed.
5. It sends a JSON Patch that tests the InferenceService's UID and resourceVersion from the preview, then adds the annotation. If the service changed since the read, the patch fails and the CLI exits 3.
6. It prints the result, an ActionResult, to stdout.

With --dry-run client, the CLI runs every step but the patch. With --dry-run server, it sends the patch with dryRun=All, so the API server and OME's webhook check the request and store nothing. Each run generates a new request ID, so a dry run's ID never becomes the real request's. Guarded actions describes the prompt, the dry-run modes and the time limit.

When the patch fails with anything but a conflict, the command exits 1. The error says outcome unknown, as in migration request failed or outcome unknown; inspect migration status using the preview UUID: followed by the cause. After a server error, a timeout or a network error, the API server may have stored the request anyway, as When the outcome is unknown explains. The CLI never retries.

Before you send a new request, look the first one up. Run start again with the same flags, the hints in the same order, and --request-id set to the Request UUID from its preview. If the request is still on the service with the same values, the lookup sends nothing and exits 0. Otherwise it refuses with the precondition conflict and exits 3: OME has taken the request, it never arrived, or your flags differ. Request an Instance migration shows how to tell which.

The preview shows the target and the request, then the component's InferenceReplica and migration mode, and the Instance's running revision and current nodes, as in the first example. Check From node and Current nodes before you confirm.

The preview ends with warnings: node hints are only preferences, and the patch guards only the InferenceService. These can join them:

The result on stdout is an ActionResult, whose request-id is the ID that kubectl ome wait takes. applied means the API server stored the request, not that OME took it or moved anything. The follow-up command shows what OME did. The table cuts long values, so use -o json or -o yaml for the full message and follow-up command.

Refusals¶

A refusal changes nothing. These come before the command reads the cluster, and exit 1:

These come after it reads the cluster, and exit 1:

start also refuses a target that's being deleted, has an unsafe identity or is too large to check safely. It refuses one that takes part in multi-cluster placement too, which is alpha, still in development, and off by default. Target checks lists these messages.

migration precondition conflicts or is stale; inspect migration status and retry explicitly means the state of the service doesn't allow the migration now, and the command exits 3. The causes, the most likely first:

• The service is paused with ome.io/rollout-paused, or its InferenceReplica is paused. See kubectl ome rollout.
• The service has the ome.io/rollout-promote or ome.io/rollout-rollback annotation. The rollback annotation stays after a rollback until a different target revision appears, as promote and rollback describes.
• The Instance doesn't exist, isn't Ready, has an operation in progress, or a rollout is moving it to a new revision. kubectl ome instance shows its state.
• The Instance already has a request waiting, or is the source or the replacement in a migration that hasn't ended.
• The component's lifecycle.migrationPolicy.mode is Never.
• The Instance's pods run on more than one node and you didn't pass --from-node, or the --from-node node hosts none of them.
• The InferenceReplicas' status lags a recent change.
• The component lacks an InferenceReplica, or another component uses OMENative and this one doesn't.
• A pod of the Instance is missing, unscheduled or not on the running revision.
• The source's node affinity requires the source node, so no replacement could run anywhere else.
• A hint names the node the CLI took as the source.
• With --request-id, the request's annotation isn't on the service, or its values differ from your flags.
• The InferenceReplica, the Instance's pods or its ControllerRevision changed after the CLI first read them.

guarded migration rejected; inspect migration status using preview UUID means the InferenceService changed after the CLI read it, often through a status update. Nothing changed, and the command exits 3. Run it again: it reads the service again, and generates a new request ID.

Examples¶

Run every check and see the request, without sending it:

kubectl ome migration start chat -n prod --component engine --instance 3 --reason "node-a maintenance" --dry-run client --yes

The preview, on stderr:

ALPHA guarded migration preview (not controller convergence)
FIELD              VALUE
Action             migration start
Context            prod-us-east
Workload NS        prod
OME NS             ome
Target             InferenceService/chat
UID                5f0c8a52-3b1e-4c8d-9f27-6a1d0e4b7c93
ResourceVersion    48213
Dry-run            client
Request UUID       6ff01fa3-2a47-4725-9c85-786ed59ac166
Schema             v1
Component          engine
Instance           3
From node          node-a
Hint nodes         <absent>
Reason             "node-a maintenance"
Requested at       2026-09-28T10:12:04Z
Requested by       kubectl-ome
Lookup only        false
Source IR          chat-engine
IR UID             0b6e2f41-9d3c-4a57-8e12-c4f5a6b7d890
IR version         48190
Migration mode     Auto
Running revision   chat-engine-5d8f7c9b
Incarnation        1
Current nodes      node-a
Node hints are soft preferences; capacity, scheduling and controller convergence
are not guaranteed.
Parent UID/resourceVersion CAS is not an IR/Pod/runtime transaction.

The result, on stdout:

FIELD           VALUE
action          migration start
target          InferenceService/prod/chat
dry-run         client
accepted        No
applied         No
request-id      6ff01fa3-2a47-4725-9c85-786ed59ac166
revision-hash   -
message         Validated locally; no patch sent.
follow-up       kubectl ome migration status chat --component=engine ...
hint            Use -o json or -o yaml for full values.

Instance 3's pods run only on node-a, so without --from-node the CLI takes node-a as the source. Nothing changed.

Move Instance 3 off node-a, preferring two nodes for the replacement, and confirm at the prompt:

kubectl ome migration start chat -n prod --component engine --instance 3 --hint-node node-c,node-d --reason "node-a maintenance"

The preview shows Hint nodes as node-c, node-d. After you answer y, the result, on stdout:

FIELD           VALUE
action          migration start
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      7c9e6679-7425-40de-944b-e07fc1f90ae7
revision-hash   -
message         API accepted annotation request; delivery and converg...
follow-up       kubectl ome migration status chat --component=engine ...
hint            Use -o json or -o yaml for full values.

Keep the request-id: OME tracks the request by it.

Look up a request whose outcome is unknown, with the flags you sent it with and the Request UUID from its preview:

kubectl ome migration start chat -n prod --component engine --instance 3 --hint-node node-c,node-d --reason "node-a maintenance" --request-id 7c9e6679-7425-40de-944b-e07fc1f90ae7

The preview shows Lookup only as true, and ends at IR version. The result, on stdout:

FIELD           VALUE
action          migration start
target          InferenceService/prod/chat
dry-run         none
accepted        No
applied         No
request-id      7c9e6679-7425-40de-944b-e07fc1f90ae7
revision-hash   -
message         Identical retained mailbox observed; no replay sent; ...
follow-up       kubectl ome migration status chat --component=engine ...
hint            Use -o json or -o yaml for full values.

The request is still on the service, waiting for OME, so the command sent nothing and exited 0. Don't send it again: run status until the request shows up.

Exit codes¶

Errors print to stderr as error: <message>. Exit codes in Guarded actions says what to do after each.

Related guides¶

• Request an Instance migration
• Let Alfred migrate Instances
• Troubleshoot an InferenceService