kubectl ome instance Unreleased: coming in v1.3

kubectl ome instance shows the Instances of an InferenceService's OMENative components. An Instance is one replica of a component: one pod, or a leader pod and its workers, which OME updates, repairs and reports as one unit. Use list to find an Instance that isn't Ready, status to see its pods and their warnings, and retry-blocks to see which revisions OME is retrying or holding, and why. Once you've fixed the cause, the alpha release-held lets OME try a held revision again.

OME reports the Instances in the status of the InferenceReplica it creates for each component, named <InferenceService>-<component>, such as chat-engine. Only components with such an InferenceReplica appear: Opt in to OMENative shows how a component gets the mode. A standalone InferenceReplica — one you create directly — appears in no report, even when an InferenceService fronts it: see Which replicas the command reads.

kubectl ome instance SUBCOMMAND INFERENCESERVICE [flags]

status, retry-blocks and release-held act on one component, which --component names: engine, decoder or router. -o json and -o yaml print the whole report, as an InstanceListReport, InstanceStatusReport, InstanceRetryBlocksReport or ActionResult. Only status has a wide view: release-held treats -o wide as table. The overview covers the other output formats and the permissions each command needs.

list and status say how current the status they show is, with an evidence level:

Which replicas the command reads¶

Every subcommand takes an InferenceService, never an InferenceReplica. It reads the InferenceService first, so naming a standalone replica — an InferenceReplica you create directly, with no parent InferenceService — fails:

kubectl ome instance list pool-a -n prod

error: get InferenceService "prod/pool-a": inferenceservices.ome.io "pool-a" not found

The subcommands then keep only the InferenceReplicas that record the InferenceService as their parent: spec.parentRef names it, the ome.io/inferenceservice label carries its name, and their controller owner reference points at it. OME writes all three on every InferenceReplica it creates, so those appear. A replica that claims none of them isn't shown and raises no issue, whatever its name. Only a replica that claims the InferenceService but fails the identity checks is reported as left out, as REJECTED or ID_REJECT.

A standalone replica claims no InferenceService, so no subcommand shows its Instances. Read them from the replica itself: in kubectl get inferencereplica pool-a -n prod -o yaml, they're the rows of status.instanceStatuses, or of status.instanceStatusColumns when status.instanceStatusEncoding is ColumnarV2. kubectl ome get ir finds standalone replicas: their PARENT column shows -.

An InferenceService whose spec.replicaRefs fronts standalone replicas writes nothing on them, so they don't claim it, and it has no InferenceReplicas of its own. Its report is empty while the fronted pools serve:

kubectl ome instance list front -n prod

COMP   IDX/INC   PHASE   PODS   REVS   AOF   EVIDENCE
-      -         -       -      -      ---   OK

The empty table isn't an error: the command read everything, found no InferenceReplica that claims front, and exits 0. status and retry-blocks read the same collection, so they can't see the fronted replicas either. To inspect a fronted pool's Instances, read the replica itself, as above.

list¶

kubectl ome instance list INFERENCESERVICE [flags]

list shows one row for each Instance of the InferenceService's OMENative components, engine first, then decoder and router, in index order. The rows come from the InferenceReplicas' status, so they show what OME last recorded. status shows the live pods.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

EVIDENCE shows one value for each component:

A component with no rows to show gets one row of dashes with its EVIDENCE, such as NOT-OBSERVED. A problem with the whole collection adds a last row, summary. It shows SOURCE-UNAVL (the CLI couldn't list or decode an InferenceReplica), REJECTED (it left out an InferenceReplica that failed the identity checks) or TRUNCATED (there's more than the table shows). With no rows at all, the table shows one row of dashes with a summary value or the overall state: OK, PARTIAL or UNAVAILABLE.

-o json and -o yaml print every value in full, and every issue behind the EVIDENCE values.

Examples¶

List the Instances of chat in prod:

kubectl ome instance list chat -n prod

COMP     IDX/INC   PHASE   PODS    REVS          AOF   EVIDENCE
engine   0/1       Ready   1/1/1   chat...ne-a   A--   OK

The engine has one Instance, index 0 in its first incarnation. It's Ready on the revision chat-engine-a, cut to 11 characters, with its one pod serving and available.

When the InferenceService also has a decoder that's updating, and OME hasn't caught up with its latest change, the table looks like this:

COMP      IDX/INC   PHASE      PODS    REVS          AOF   EVIDENCE
engine    0/1       Ready      1/1/1   chat...ne-a   A--   OK
decoder   3/8       Updating   1/1/4   chat...-new   AO-   STALE:3/4

The decoder's Instance 3 is in its eighth incarnation and moving from chat-decoder-old to chat-decoder-new, with an operation under way. One of its four pods is serving and available. STALE:3/4 means OME last observed generation 3 of the decoder's InferenceReplica, which is now at generation 4.

status¶

kubectl ome instance status INFERENCESERVICE INDEX --component COMPONENT [flags]

status shows one Instance: the row OME reports for it, the migrations it takes part in, and its pods with their Warning events. INDEX is the Instance's index, the first number under IDX/INC in list. The pod and event rows are live. The phase, revisions, pod counts, conditions, operation and last failure are what OME last recorded.

status first works out the component's deployment mode, from the ome.io/deploymentMode annotation or else, as OME does, from the InferenceService and its runtime. A component in another mode gets the state NotOMENative.

status finds the Instance's pods by their ome.io/instance-index label. When an Instance has more than 16 pods, it shows the 16 most likely to explain a problem, such as pods being deleted or not Ready. It shows the reason and count of each pod's Warning events, but not their messages: kubectl describe pod shows those.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The first row, state, gives the report's state, the Instance and the component's evidence level, as in Reported engine[0] evidence=Reported. The state is one of:

The deployment and encoding rows always show. The others appear when they have a value:

The op hold row's waiting= shows only a token the CLI recognizes: QuotaExceeded, NodeUnknown, SourceUnrouted, Unschedulable, PodGroupTerminating or Paused. OME holds operations on tokens outside that list, such as CapacityProvisioning, while an Instance's pods wait for their TPU slice, and AdmissionUnavailable, while the API server can't reach an admission webhook for the operation's pod writes. The CLI shows those, like any token it doesn't know, as waiting=Unknown, and adds an issue row OperationWaitingUnknown. The report itself carries the normalized value, so -o wide, -o json and -o yaml show Unknown too. The real token isn't lost: both holds also record the Instance's last failure under it, so the failure row's reason= names the hold, and the InferenceReplica's own status keeps the raw waiting. The op strategy row works the same way: it shows SurgeThenDrain, RecreatePod, InPlaceIfPossible or InPlaceOnly, and any other value as Unknown, with an issue row OperationStrategyUnknown.

The default table cuts long values, so the op timing, failure, migration and pod rows often lose their last fields. -o wide prints every field in full, one to a row. In a pod row, ready is the pod's Kubernetes Ready condition, and serving is the ome.io/serving readiness gate that OME sets. deleting is true when the pod has a deletion timestamp: Recover stuck deletions covers pods that stay that way. Text that looks like a credential shows as [REDACTED].

A migration request can wait in the Accepted phase, and its deadline keeps running while it waits, even while the rollout is paused. kubectl ome migration inspects and requests migrations.

Examples¶

Show Instance 0 of the engine of chat:

kubectl ome instance status chat 0 --component engine -n prod

FIELD        VALUE
state        Reported engine[0] evidence=Reported
deployment   mode=OMENative source=ServiceSpec origin=LiveRuntime evidence...
encoding     name=DenseV1 evidence=Reported reason=-
instance     chat-engine inc=1 phase=Failed admitted=true
revisions    running=- target=chat-engine-6c4f8b1d
persisted    pods=1 serving=0 available=0
lifecycle    activeOrdinal=0 readySince=-
failure      pod=chat-engine-0-default-0 container=ome-container reason=Er...
fail time    2026-09-14T20:52:31Z
pod          chat-engine-0-default-0 default/Running ready=False serving=F...
event        Pod/chat-engine-0-default-0 reason=BackOff count=21

Instance 0 is Failed and has never been Ready, so revisions names only its target. OME recorded its last failure at 2026-09-14T20:52:31Z, and its pod has a BackOff event with a count of 21. Add -o wide to see the cut failure and pod fields, such as failure exit code and pod[0] restarts.

retry-blocks¶

kubectl ome instance retry-blocks INFERENCESERVICE --component COMPONENT [flags]

retry-blocks shows the revisions that OME is retrying or holding for one component, from its InferenceReplica's status.retryBlocks. A revision here is a version of the component's pod template, named <InferenceService>-<component>-<hash>, such as chat-engine-7f9c4d2b, not a runtime revision. When updates to a revision fail, OME retries it until its attempts run out, then holds it. Only failures that the revision causes, such as an image that can't be pulled, count as attempts. A held revision stays held until you release it with release-held or change the component's pod template. How a revision gets held lists the failures that count and the retry defaults.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The digest isn't the revision's hash, so don't pass it to release-held. Take the full name from -o json, or use its hash, the last 8 characters of the name.

REL is YES only for a valid Held block on current status, while nothing is being deleted. release-held checks more, so it can still refuse a block marked YES, as Refusals lists.

Each issue adds a row with ISSUE in STATE and an alias in REASON:

A component with no blocks and no issues shows one row with EMPTY in STATE.

-o json and -o yaml print each block in full, with releaseEligible for REL, and a summary with the block counts.

Examples¶

Show the retry blocks of the engine of chat:

kubectl ome instance retry-blocks chat --component engine -n prod

COMP     STATE   TARGET           ATT   NEXT   REL   REASON
engine   HELD    chat-#4c6a152e   3     -      YES   ImagePullB...

The engine holds a revision after 3 attempts. -o json shows its full name, chat-engine-7f9c4d2b, and the full reason, ImagePullBackOff. REL shows that the block qualifies for release-held, either by that name or by the hash 7f9c4d2b.

release-held¶

kubectl ome instance release-held INFERENCESERVICE [flags]

release-held asks OME to release one Held retry block of a component, so that OME tries the revision again. It sets the ome.io/release-held-revision annotation on the component's InferenceReplica to the block's full target revision. OME then removes the Held block, deletes the annotation and goes back to updating the component's Instances to the revision, so fix what made the revision fail first. A paused InferenceService stays paused, and the update waits until you resume it. The command doesn't wait for OME: check with retry-blocks, or in a script with kubectl ome wait --for=held-revision=unheld. Release a held revision walks through a release.

--revision takes the block's full target revision, such as chat-engine-7f9c4d2b, or its hash, the last 8 characters, such as 7f9c4d2b. Either way, the annotation gets the full name. Besides the read permissions, release-held needs patch on inferencereplicas.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

How release-held runs¶

release-held follows the guarded flow of the rollout actions, with the component's InferenceReplica as its target. Its preview, a HELD RELEASE PREVIEW table on stderr, shows the revision with its state and attempts, and the annotation it sets. After you confirm, it reads everything again, even for a client dry run. It has 45 seconds in all, including the time the prompt waits for you. These errors come after you confirm:

After any of these errors, run retry-blocks before you try again.

The result depends on the dry-run mode:

applied means that the API server stored the annotation, not that OME released the block. --yes skips only the prompt, never a check. Guarded actions describes the preview, the prompt, the dry-run modes and the ActionResult.

Refusals¶

Like every guarded action, release-held refuses an InferenceService that's being deleted, has an unsafe identity, is too large to check safely, or uses multi-cluster placement (alpha). Target checks lists those messages. It also refuses, changes nothing and exits 1 in these cases:

The first refusal has these causes, likeliest first:

• No block matches --revision, or the block that matches isn't Held.
• An InferenceReplica of the InferenceService, even another component's, has stale status. Wait for OME to catch up.
• A block on the component's InferenceReplica is malformed, as when its failure time is in the future because your machine's clock runs behind the cluster's.
• An InferenceReplica of the InferenceService fails the identity checks or is being deleted, or the component's InferenceReplica lacks ome.io/controller-write: "true".
• The component doesn't run in OMENative, has no InferenceReplica, or is claimed by two InferenceReplicas.

The command refuses the release shows how to find out which applies.

Examples¶

Run every check and see the preview, without sending the patch:

kubectl ome instance release-held chat --component engine --revision chat-engine-7f9c4d2b -n prod --dry-run client

The preview goes to stderr. After you confirm, the result goes to stdout:

FIELD           VALUE
action          instance release-held
target          InferenceReplica/prod/chat-engine
dry-run         client
accepted        No
applied         No
request-id      -
revision-hash   7f9c4d2b
message         Validated locally; no patch sent.
follow-up       kubectl ome instance retry-blocks chat --component=en...
hint            Use -o json or -o yaml for full values.

Release the revision by its hash, without a prompt, as a script would:

kubectl ome instance release-held chat --component engine --revision 7f9c4d2b -n prod --yes

FIELD           VALUE
action          instance release-held
target          InferenceReplica/prod/chat-engine
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   7f9c4d2b
message         API accepted release annotation request; controller r...
follow-up       kubectl ome instance retry-blocks chat --component=en...
hint            Use -o json or -o yaml for full values.

Step 2 of the guide shows the result as JSON, with every value in full.

Exit codes¶

Errors print to stderr as error: <message>.

Related guides¶

• Release a held revision
• Reset failed Instances
• Set Instance readiness deadlines
• Change the Instance status encoding
• Request an Instance migration
• Request a transient scale
• Serve a model on OMENative
• Serve a prefill-decode model
• Troubleshoot an InferenceService