kubectl ome rollout Unreleased: coming in v1.3

kubectl ome rollout shows where an InferenceService is in a rollout, and steers a rollout in progress. The read commands show progress, the plan a run follows and recent runs, and check the rollout, traffic and autoscaling configuration. The alpha actions pause and resume rollout work, promote or roll back a canary, and repin a drifted plan. Rollout groups and alpha rollout policies describe the plans. Rollout groups and the actions need components that use OMENative.

kubectl ome rollout SUBCOMMAND INFERENCESERVICE [flags]

Each subcommand takes the name of one InferenceService and prints a table. For the read commands, -o wide prints more detail. -o json and -o yaml print the whole report, including its warnings, as Output formats describes. Required RBAC lists the permissions each command needs.

The reports mark how the CLI got each value with an evidence level.

status¶

kubectl ome rollout status INFERENCESERVICE [flags]

status shows where each component of the InferenceService is in its rollout: its rollout group, its phase, the current step of a canary, and the revisions it serves. The values come from the controller's status. While a run is active, the steps come from the plan the run pinned.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has a SERVICE column for the whole InferenceService, then a column for each component: ENGINE, DECODER and ROUTER. The first rows describe the service:

A service with rollout groups, or with rollout phases or revisions in its status, shows STATE Unknown. The CLI can't tie that status to the current generation, so EVIDENCE shows Reported, EPOCH shows Unverifiable, and ISSUES lists EpochUnverifiable. Read REPORTED for the state the status records.

Staged means a group finished rolling out up to the rollingUpdate.partition its components set, which holds that many instances on the old revision. NotConfigured means the service doesn't use rollouts.

A component row appears only when a component has a value for it:

In a canary group, ROLLED-OUT, READY and PREVIOUS don't change, even after the canary finishes. They keep the revisions from before the component joined the group, or stay empty if it started in one. The canary's own revisions show in explain, as stable= and target=.

STRATEGY is one of:

GATE is one of:

Canary progression describes how a step advances, and Canary metric analysis the analysis checks.

Before the last step, a canary waiting at any gate, or holding after a repin, is Paused, and so is REPORTED. At the last step, the canary waits as Promoting, so REPORTED stays InProgress.

The last row, ISSUES, lists issue codes for the service and for each component, in their own columns. An issue tied to a rollout group shows as Code(group=N). Any issue also adds a PartialData warning. The issues are:

-o wide prints a row per component, with every field but COORDINATION as a column.

Examples¶

Show the rollout of chat in prod while its canary waits at a manual gate:

kubectl ome rollout status chat -n prod

FIELD          SERVICE             ENGINE       DECODER   ROUTER
STATE          Unknown             -            -         -
REPORTED       Paused              -            -         -
EVIDENCE       Reported            -            -         -
EPOCH          Unverifiable        -            -         -
COORDINATION   NotApplicable       -            -         -
GROUP          -                   0            -         -
STRATEGY       -                   Canary       -         -
GROUP-PHASE    -                   Paused       -         -
PHASE          -                   Paused       -         -
STEP           -                   1/2          -         -
GATE           -                   Manual       -         -
CAPACITY       -                   25%          -         -
TRAFFIC        -                   10% -> 10%   -         -
ISSUES         EpochUnverifiable   -            -         -

The engine's canary waits at the first of two steps, with 25% of its capacity on the new revision and a recorded traffic weight of 10%. PHASE is Paused and GATE is Manual, so it waits for a promote.

For a service whose engine is in no rollout group, print the report as JSON:

kubectl ome rollout status chat -n prod -o json

{
  "apiVersion": "cli.ome.io/v1alpha1",
  "kind": "RolloutStatusReport",
  "metadata": {
    "namespace": "prod",
    "name": "chat"
  },
  "collectedAt": "2026-08-31T18:30:00Z",
  "sources": [
    {
      "kind": "InferenceService",
      "namespace": "prod",
      "name": "chat",
      "uid": "isvc-uid",
      "generation": 7,
      "evidence": "Observed",
      "collectedAt": "2026-08-31T18:30:00Z"
    }
  ],
  "content": {
    "summary": {
      "state": "Unknown",
      "reportedState": "Succeeded",
      "evidence": "Reported",
      "epoch": "Unverifiable",
      "coordinationReady": "NotApplicable"
    },
    "groups": [],
    "components": [
      {
        "type": "engine",
        "strategy": "Independent",
        "phase": "Stable",
        "traffic": []
      }
    ],
    "issues": [
      {
        "code": "EpochUnverifiable"
      }
    ]
  },
  "warnings": [
    {
      "code": "PartialData"
    }
  ]
}

The engine's current and update revisions match, so the CLI works out Stable, and reportedState is Succeeded.

explain¶

kubectl ome rollout explain INFERENCESERVICE [flags]

explain shows the rollout plan as you declared it, as it resolves now, and as the controller runs it, next to the progress the controller reports. Use it to see which plan a run follows and what holds a rollout.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has a section for each view of the plan:

Each view lists these items:

The Effective view also shows the plan conditions, the holds and the observed progress:

READY is False with the reason PolicyNotFound, PolicyNotReady, ProgressionMismatch, PlanInvalid or ProviderUnbound. READY and DRIFT show Unobserved; evidence=Unavailable until the controller sets the condition. They show Invalid/Unknown, and ISSUES lists PlanConditionMalformed, when the condition is malformed or conflicts with whether a run is active.

The source in a GROUP row is one of:

CONFIG lists a group's settings by key:

The last four settings print as value(Source;Effect), as in soak=1m0s(Configured;IgnoredFinalGroup). A setting that takes effect leaves out the effect, as in surge=25%(Defaulted). One the CLI can't resolve leaves out the value and the parentheses, as in ratio=Unresolved.

HOLD is one of:

-o wide prints a row per canary step, or per group without steps, with the group's fields on each row. A component in no rollout group gets a row with VIEW Observed.

Examples¶

Explain the rollout of chat in prod:

kubectl ome rollout explain chat -n prod

VIEW        ITEM        DETAIL
Declared    PLAN        groups=1
            GROUP 0     Canary; components=engine; source=Declared/Inline
            STEP 1/2    capacity=25%; traffic=10%; gate=Manual
            STEP 2/2    capacity=100%; traffic=100%; gate=Immediate
Live        PLAN        mode=Live; groups=1
            GROUP 0     Canary; components=engine; source=Declared/Inline
            STEP 1/2    capacity=25%; traffic=10%; gate=Manual
            STEP 2/2    capacity=100%; traffic=100%; gate=Immediate
Effective   PLAN        mode=Pinned; evidence=Reported; groups=1
            READY       True/Pinned; evidence=Reported
            DRIFT       True/SpecNewerThanRun; evidence=Reported
            GROUP 0     Canary; components=engine; source=Reported/Policy
            CONFIG      policy=RolloutPolicy/guarded@4
                        digest=rp1:aaaaaaaaaaaa
            PHASE       Canarying
            REVISIONS   stable=aaaaaaaa,target=bbbbbbbb
            TRAFFIC     engine:aaaaaaaa=80%,engine:bbbbbbbb=20%
            STEP 1/2    capacity=50%; traffic=20%; gate=Manual
            STEP 2/2    capacity=100%; traffic=100%; gate=Immediate
            ISSUES      EpochUnverifiable

The run pinned its plan from the rollout policy guarded at generation 4. That plan puts half the capacity and a 20% traffic weight on the new revision until a manual promote, then all of it. The spec now declares a quarter of the capacity and a 10% weight, so DRIFT is True/SpecNewerThanRun.

history¶

kubectl ome rollout history INFERENCESERVICE [flags]

history shows the rollout records the InferenceService keeps in its status: the active run, the last closed run, where each group's plan came from, and the current revisions. The status keeps only those two runs, so history isn't an audit trail.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

Each row is one record, and TYPE says what kind:

TIME is in UTC, and ISS counts the issues in the whole report. In the PROV rows, DETAIL names the policy the group uses or shadows. STATE shortens long values, as in NotConf... for NotConfigured. A policy name over 10 characters prints as its first character, # and a hash. REV shortens four phases: BGStandby is BlueGreenStandby, RollingBk is RollingBack, ScaleDown is ScalingDown, and AwaitNext is AwaitingNextComponent.

-o wide prints every kept field in full, one row per record.

Examples¶

Show the kept history of chat in prod:

kubectl ome rollout history chat -n prod

TYPE     STATE       COMP     IDENT          TIME          DETAIL       ISS
WINDOW   Partial     -        -              -             bounded      1
CURR     Unknown     -        Reported       -             Unverified   1
ACTIVE   Active      -        0123456789ab   08-31T18:00   G:1 T:1      1
TARGET   Reported    engine   cccccccc       -             run-target   1
LAST     Completed   -        -              08-31T17:00   G:1          1
PROV-A   Inline      g0       cca45ec1fb0f   08-31T18:30   -            1
PROV-L   Inline      g0       cca45ec1fb0f   08-31T17:00   -            1
PROV-C   Inline      g0       cca45ec1fb0f   -             -            1
REV      Unknown     engine   aaaaaaaa       -             Current      1
REV      Unknown     engine   bbbbbbbb       -             Ready        1
REV      Unknown     engine   dddddddd       -             Previous     1

The active run 0123456789ab opened at 18:00 UTC with one group and one target, engine revision cccccccc. The run before it completed at 17:00. All three plan records show the same inline plan.

validate¶

kubectl ome rollout validate INFERENCESERVICE [flags]

validate checks the rollout, traffic and autoscaling configuration of an InferenceService, and the controller's evidence that parts of it took effect. It prints the report, then exits 0 when OVERALL is Valid, 2 when OVERALL is Invalid or Unverifiable, and 1 when it can't run.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

Each row is one check. COMP names the component for a per-component check. RESULT is Valid, Invalid, Unverifiable or NotApplicable, and SOURCE shows the evidence and its freshness, as in Computed/Current. The freshness is Current, Stale, Unverifiable or NotApplicable.

OVERALL is Invalid when any check is, otherwise Unverifiable when any check is, and otherwise Valid. When OVERALL is Invalid, validate then prints error: rollout validation found invalid configuration to stderr. When it's Unverifiable, it prints error: rollout validation could not verify all prerequisites.

AUTOSCALER-SPEC's service row checks the ome.io/autoscalerClass, ome.io/metrics and ome.io/targetUtilizationPercentage annotations. Each component's row checks its autoscaler block, its replica bounds and its autoscalerPolicyRef. Rollout groups, Canary progression and Component autoscaling list what admission rejects in these fields.

ROLLOUT-RESOLVE, TRAFFIC-READY and AUTOSCALER-RESOLVE check the controller's evidence. They report any problem as Unverifiable, including a plan or autoscaler policy that fails to resolve, and -o wide shows the cause.

To see whether the pinned plan still matches the spec, check DRIFT in explain.

-o json and -o yaml also list warnings: PartialData when OVERALL is Unverifiable, and StaleEvidence when any check's freshness is Stale.

-o wide splits SOURCE into EVIDENCE and FRESHNESS, and adds each check's issue codes.

Examples¶

Validate chat in prod:

kubectl ome rollout validate chat -n prod

CHECK             COMP     RESULT          SOURCE
OVERALL           -        Valid           Computed/Current
ROLLOUT-REFS      -        Valid           Computed/Current
ROLLOUT-PLAN      -        Valid           Computed/Current
ROLLOUT-ORDER     -        Valid           Computed/Current
ROLLOUT-RESOLVE   -        NotApplicable   Computed/NotApplicable
TRAFFIC-SPEC      -        Valid           Computed/Current
TRAFFIC-READY     -        NotApplicable   Computed/NotApplicable
SCALING-POLICY    -        Valid           Computed/Current
AUTOSCALER-SPEC   -        Valid           Computed/Current
AUTOSCALER-SPEC   engine   Valid           Computed/Current

ROLLOUT-RESOLVE and TRAFFIC-READY are NotApplicable, because this InferenceService has no rollout groups, active run or traffic settings. Every other check is Valid, and validate exits 0.

pause and resume¶

kubectl ome rollout pause INFERENCESERVICE [flags]
kubectl ome rollout resume INFERENCESERVICE [flags]

pause holds rollout work across the whole InferenceService by setting the ome.io/rollout-paused annotation to true. resume removes the annotation, which has two depths:

pause sets true; set freeze with kubectl annotate. A pause covers the OMENative lifecycle of the whole service. Deliberate scale-down and deletion still proceed. Timed canary gates keep aging, so a canary can advance as soon as you resume. Pause and resume a rollout shows both commands in use.

resume acts only on the pause. A canary that ended Failed or RolledBack holds until an operator acts, whether or not the service is paused: on a paused service resume ends the pause and the canary stays parked, and on a service that isn't paused it refuses with action refused: service has no recognized pause. What re-enters a parked canary is the ome.io/rollout-resume annotation, a separate controller verb that no kubectl ome command sets. Set it with kubectl annotate to the parked canary revision hash, from status.components.<component>.canary.canaryRevisionHash. OME clears the terminal phase, restarts the canary at its first step on that same revision with every gate running again, and removes the annotation once it acts. The service has no recognized pause shows the commands.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

How the actions run¶

pause, resume, promote, rollback and repin change the InferenceService the same guarded way:

1. The CLI checks the flags, reads what the action needs, and refuses if the action isn't safe now. A refusal changes nothing and exits 1.
2. It prints a preview of the exact change to stderr: the context, the namespaces, the target with its UID and resourceVersion, the annotation, and warnings.
3. It asks Confirm this exact action? [y/N] on stderr, and goes ahead only on y or yes. --yes confirms without the prompt, and scripts need it.
4. It sends a JSON Patch that tests the UID and resourceVersion from the preview. If the InferenceService changed since the read, the patch fails and the CLI exits 3.
5. It prints the result, an ActionResult, to stdout.

Any other answer, or a missing terminal, fails with action not confirmed; noninteractive input requires --yes. The command has 45 seconds in all, including the prompt, and fails with context deadline exceeded when they run out.

--dry-run client runs the checks and the prompt, then stops before the patch. --dry-run server sends the patch with dryRun=All, so the API server checks it and stores nothing.

applied in the result means the API server stored the annotation, not that the controller acted on it. Run the follow-up command to see what the controller did.

If the patch fails with anything but a refusal or a conflict, it may have been stored, so check status before you run the command again. Guarded actions describes the whole contract, including unknown outcomes and the ActionResult.

Every action also refuses a target that's being deleted, has an unsafe identity, or exceeds the CLI's inspection bounds. The same goes for a target in multi-cluster placement, which is alpha and still in development. Target checks lists these messages.

Among the reads in step 1, pause, resume, promote and rollback resolve the runtime, and when spec.runtime.name is set, the runtime revisions the InferenceService names: the ControllerRevisions in the --ome-namespace namespace that spec.runtime.revision and status.pinnedRevisionName name. Every one of those reads must succeed. A named revision that doesn't exist or can't be read refuses the action with required Kubernetes API request failed; check access and connectivity and exit code 1, before the preview, in every dry-run mode and with --yes. repin reads only the InferenceService, so these reads don't apply to it.

That message is the same fixed text as for any failed API request, so it can appear while the API server is healthy: a pinned revision that was deleted, a ControllerRevision read that RBAC denies, or the wrong --ome-namespace all refuse this way, even when the service follows the live runtime with autoSync. The read-only kubectl ome runtime effective reports the same failed read as a RevisionNotFound or RevisionUnavailable issue instead of refusing, so run it to tell a missing revision from a connectivity problem.

Refusals¶

pause and resume also refuse in these cases:

Examples¶

Pause chat in prod:

kubectl ome rollout pause chat -n prod

The command prints its preview to stderr and asks for confirmation. After your y, the result is:

FIELD           VALUE
action          rollout pause
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   -
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

Resume without a prompt, as a script would:

kubectl ome rollout resume chat -n prod --context=prod-cluster --yes

ALPHA guarded action preview (not controller convergence)
FIELD               VALUE
Action              rollout resume
Context             prod-cluster
Workload NS         prod
OME NS              ome
Target              InferenceService/chat
UID                 5d9f3c2a-8b41-4f7e-a0c6-1e2b3c4d5e6f
ResourceVersion     184467
Dry-run             none
Affected            engine
Pause depth         true
Remove annotation   ome.io/rollout-paused
Value               "true"
Active records      operations=0 migrations=0
Scope: service-wide OMENative lifecycle, not a full workload freeze.
true holds Update/Create/Migration; RestartPolicy repair continues.
freeze also holds existing-instance repair; resume clears either depth.
Canary timed gates continue aging and may advance immediately on resume.
Deliberate scale-down and deletion teardown can still proceed.
Parent UID/resourceVersion CAS is not a multi-object transaction.
FIELD           VALUE
action          rollout resume
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   -
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

The preview comes first, on stderr, from the ALPHA line to the last warning. The table after it is the ActionResult, on stdout.

promote and rollback¶

kubectl ome rollout promote INFERENCESERVICE [flags]
kubectl ome rollout rollback INFERENCESERVICE [flags]

promote and rollback act on the canary group of the active run, at the step the canary has reached.

promote sets the ome.io/rollout-promote annotation to the canary revision hash, which passes the manual gate the canary waits at. Each promote advances one step. The controller removes the annotation once it records the advance, and another promote refuses until then. With --override-analysis --yes, it passes an analysis gate instead, for this step only, skipping its checks, warm-up and bake.

rollback sets the ome.io/rollout-rollback annotation to true. It works at any step while the canary is Pending, Canarying, Paused, Promoting or Failed. The controller returns every member of the canary group to its own stable revision, and the rollout spec stays as it is.

The group stays on its stable revisions until a different target revision appears, and the controller removes the rollback annotation only then. While the annotation is there, pause, promote, rollback and repin refuse, and so do kubectl ome scale, kubectl ome runtime sync and kubectl ome migration start. On a paused service, resume --discard-pending-actions --yes removes the annotation. To remove it yourself, run:

kubectl annotate inferenceservice chat -n prod ome.io/rollout-rollback-

inferenceservice.ome.io/chat annotated

Removing the annotation doesn't retry the rejected revision.

Both commands follow the guarded flow. Their preview adds the canary's run, phase, step, gate, traffic weights and revisions, so you can check the step you act on. It counts steps from 0, as in 0 of 2 (zero-based), while status and explain count from 1. At an analysis step, it also shows the analysis state and each metric's results. Promote or roll back a canary shows both commands in use.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Refusals¶

Besides the refusals every action shares, promote and rollback refuse in these cases:

promote passes the gate only when all of these hold:

• The gate matches the flag: Manual for plain promote, Analysis with --override-analysis --yes. Timed and immediate gates advance on their own.
• Before the last step, the canary is Paused. At the last step, it's Promoting, and the step's traffic weight is 100%.
• The controller recorded when the canary entered the step, and both sides of TRAFFIC match.
• The controller finished the previous promote.

It also refuses while the canary holds after a repin, which explain shows as HOLD CanaryPreStep. Repin a drifted rollout plan shows how to release the hold.

Examples¶

These examples run in a kubeconfig context named prod-us-east. Promote a canary waiting at a manual gate:

kubectl ome rollout promote chat -n prod

After the preview and your y, the result is:

FIELD           VALUE
action          rollout promote
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

Promote past an analysis gate, skipping its checks for this step:

kubectl ome rollout promote chat -n prod --override-analysis --yes

FIELD           VALUE
action          rollout promote
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         Analysis override: API accepted annotation request; c...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

Roll back the canary:

kubectl ome rollout rollback chat -n prod

After the preview and your y, the result is:

FIELD           VALUE
action          rollout rollback
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

In each result, revision-hash is the canary revision. After a rollback, it's the revision the controller rejects.

repin¶

kubectl ome rollout repin INFERENCESERVICE [flags]

A run pins its plan when it starts. An edit to the rollout spec or a rollout policy then waits for the next run, and explain shows DRIFT. Run repin to move the active run to the current plan now. It sets the ome.io/rollout-repin annotation to the current plan's digest, and the controller repins only if its own render of the plan has that digest.

The controller removes the annotation and reports the outcome as an event on the InferenceService:

A canary keeps its step number, or takes the new plan's last step if the new plan has fewer steps. If that step's traffic weight is higher than the one last recorded, the canary holds before the step, with HOLD CanaryPreStep. When the current plan already matches the pinned plan, the controller removes the annotation without an event.

An annotation left when a run closes stays until the next run opens. Set by hand, the value now skips the digest check, as Repin by annotation shows.

The live plan must keep the pinned topology: the same number of groups, and the same progression, components, order, soak and ratio in each. Repin a drifted rollout plan shows the command in use.

repin follows the guarded flow, with these differences:

• It reads only the InferenceService, and has no --ome-namespace flag.
• Its follow-up command is kubectl ome rollout explain.
• Its ActionResult shows run-id, pinned-plan-digest, requested-plan-digest and group-count in place of request-id and revision-hash.
• It reports a server error, a timeout or a network error as an unknown outcome, as in rollout repin request outcome unknown; do not replay, check rollout explain. Check explain before you run repin again.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Refusals¶

Besides the refusals every action shares, repin refuses in these cases:

Examples¶

Have the API server check the repin without storing it:

kubectl ome rollout repin chat -n prod --dry-run server --yes

FIELD                   VALUE
action                  rollout repin
target                  InferenceService/prod/chat
dry-run                 server
accepted                Yes
applied                 No
run-id                  chat-0123456789ab
pinned-plan-digest      rp1:56fe0ddd6edd
requested-plan-digest   rp1:1d153db2a5df
group-count             1
message                 API dry-run accepted; no changes persisted.
follow-up               kubectl ome rollout explain chat -n prod --context=pr...
hint                    Use -o json or -o yaml for full values.

Repin, confirming at the prompt:

kubectl ome rollout repin chat -n prod

After the preview and your y, the result is:

FIELD                   VALUE
action                  rollout repin
target                  InferenceService/prod/chat
dry-run                 none
accepted                Yes
applied                 Yes
run-id                  chat-0123456789ab
pinned-plan-digest      rp1:56fe0ddd6edd
requested-plan-digest   rp1:1d153db2a5df
group-count             1
message                 API accepted repin annotation; controller consumption...
follow-up               kubectl ome rollout explain chat -n prod --context=pr...
hint                    Use -o json or -o yaml for full values.

The request asks to move run chat-0123456789ab from plan rp1:56fe0ddd6edd to rp1:1d153db2a5df. Run the follow-up explain to see whether DRIFT returned to False/InSync.

Exit codes¶

Errors print to stderr as error: <message>.

Related guides¶

• Pause and resume a rollout
• Promote or roll back a canary
• Repin a drifted rollout plan
• Troubleshoot an InferenceService