OME
ome-projects/ome
  • v1.2.2
  • 516
  • 102
↑ ↓ to move ↵ to open esc to close See all results

Promote or roll back a canary Unreleased: coming in v1.3

Advance a gated canary step with kubectl ome rollout promote, or abort the canary back to the stable revisions with rollback; both are alpha.

kubectl ome rollout promote moves a canary past a step that waits for you, and kubectl ome rollout rollback returns the canary to its stable revision at any step. You'll also recover a canary that parks Failed, when a step never becomes ready or a canary pod keeps crashing. Both commands are alpha, so their flags and behavior can change between releases. They work on components that use the OMENative deployment mode, and Deployment modes shows how to opt in.

How a canary step gates

Each step moves capacity, a share of the component's Instances, to the new revision, and records traffic, a weight for it. The canary waits in the phase Pending until the step's new Instances are ready. Then the step's gate decides when it moves on:

Gate The step sets The step advances when To move it on yourself
Immediate Only capacity and traffic Its traffic weight is recorded. Nothing: it moves on at once.
Timed pause with a duration The duration has passed since the weight was recorded. Wait for the duration. promote refuses.
Manual pause: {} You run promote. Run kubectl ome rollout promote.
Analysis analysis Its metric checks pass. Run kubectl ome rollout promote --override-analysis --yes.

While the canary waits at a gate, its phase is Paused, or Promoting at the last step. An analysis gate can also roll the canary back on its own when too many checks fail.

The engine of chat rolls out in two steps:

YAML
spec:
  rollout:
    groups:
      - components:
          - engine
        canary:
          steps:
            - capacity: "25%"
              traffic: 10
              pause: {}
            - capacity: "100%"
              traffic: 100

The first step moves a quarter of the engine's Instances to the new revision, records a 10% weight and waits for a promote. The last step's gate is immediate, so the canary completes once every Instance runs the new revision. A canary keeps the steps it started with, so an edit to the canary block while it runs waits for the next rollout, unless you repin it.

Step 1: Check the canary's state

Check which step the canary is at and which gate holds it:

Shell
kubectl ome rollout status chat -n prod
Output
FIELD          SERVICE             ENGINE       DECODER   ROUTER
STATE          Unknown             -            -         -
REPORTED       Paused              -            -         -
EVIDENCE       Reported            -            -         -
EPOCH          Unverifiable        -            -         -
COORDINATION   NotApplicable       -            -         -
GROUP          -                   0            -         -
STRATEGY       -                   Canary       -         -
GROUP-PHASE    -                   Paused       -         -
PHASE          -                   Paused       -         -
STEP           -                   1/2          -         -
GATE           -                   Manual       -         -
CAPACITY       -                   25%          -         -
TRAFFIC        -                   10% -> 10%   -         -
ISSUES         EpochUnverifiable   -            -         -

The engine's canary is at step 1 of 2, with 25% of its Instances on the new revision. PHASE is Paused and GATE is Manual, so it waits for a promote. TRAFFIC shows the step's weight, then the weight OME recorded. Any service with rollout groups shows STATE Unknown and the issue EpochUnverifiable, so read REPORTED and the ENGINE column instead. Output fields describes each row.

For the canary's revisions, run kubectl ome rollout explain chat -n prod: REVISIONS shows them as stable= and target=.

Step 2: Pass the gate

At a manual gate

When GATE shows Manual, promote the canary. promote sets the ome.io/rollout-promote annotation to the canary's revision, and OME moves the canary one step:

Shell
kubectl ome rollout promote chat -n prod

The command previews the step, its gate and the revisions, and asks you to confirm. The preview counts steps from 0, so the first step shows as 0 of 2 (zero-based). Answer y when it shows the step you expect. In a kubeconfig context named prod-us-east, the result is:

Output
FIELD           VALUE
action          rollout promote
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

applied means that the API server stored the request, not that the canary moved. Step 3 shows how to check. -o yaml prints the values that the table cuts short.

At an analysis gate

When GATE shows Analysis, the canary advances on its own once its metric checks pass, and plain promote refuses. To pass the gate without waiting for the checks, add --override-analysis, which requires --yes. The override passes only the current step's gate, whether the analysis is warming up, sampling or baking.

Because --yes skips the prompt, run the command with --dry-run client first. It prints the preview, with the analysis state and each metric's results, and sends nothing. Then run:

Shell
kubectl ome rollout promote chat -n prod --override-analysis --yes

The preview's warning starts with ANALYSIS OVERRIDE: bypasses health checks, warm-up and bake, and the result's message starts with Analysis override::

Output
FIELD           VALUE
action          rollout promote
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         Analysis override: API accepted annotation request; c...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

Step 3: Check that the canary moved on

OME acts on the request after the command returns, so run the status command from Step 1 again. STEP shows the next step, and PHASE shows Pending or Canarying while the canary moves to it, then Paused at its next gate. For chat, STEP shows 2/2 while the rest of the Instances move to the new revision. The last step's gate is immediate, so PHASE then shows Stable and STEP shows -. When the last step pauses or runs analysis, or the group sets scaleDownDelaySeconds, the canary waits in Promoting first. OME removes the ome.io/rollout-promote annotation once it records the advance, and another promote refuses until then.

Roll back the canary

rollback returns the canary to its stable revision. It works at any step and gate, even on a Failed canary, until the canary completes or rolls back:

Shell
kubectl ome rollout rollback chat -n prod

Answer y when the preview shows the canary you mean. The result is:

Output
FIELD           VALUE
action          rollout rollback
target          InferenceService/prod/chat
dry-run         none
accepted        Yes
applied         Yes
request-id      -
revision-hash   bbbbbbbb
message         API accepted annotation request; convergence not obse...
follow-up       kubectl ome rollout status chat -n prod --context=pro...
hint            Use -o json or -o yaml for full values.

revision-hash shows the rejected canary revision. OME records a 0% weight for it at once, but its Instances keep getting requests until OME replaces them with Instances on the stable revision. In a group of several components, each component goes back to its own stable revision. PHASE shows RollingBack until only stable Instances remain, then RolledBack.

A rollback holds the rejected revision. The spec still asks for it, and explain shows it in REVISIONS as rejected=. The component stays on its stable revision until a change to the spec produces a revision other than the rejected and stable ones.

The ome.io/rollout-rollback annotation stays on the InferenceService after the rollback, but the request is spent. You don't have to remove it before you apply a fix: when a change produces such a new revision, OME starts the fresh canary toward it and removes the annotation itself, so the lingering annotation doesn't roll the new canary back. While it's there, pause, promote, rollback and repin refuse, and so do kubectl ome scale, kubectl ome runtime sync and kubectl ome migration start. To use those commands, remove it:

Shell
kubectl annotate inferenceservice chat -n prod ome.io/rollout-rollback-
Output
inferenceservice.ome.io/chat annotated

Removing the annotation doesn't retry the rejected revision. A canary that its analysis rolled back holds the revision the same way, but without the annotation.

When a step never becomes ready

When a step's new Instances aren't ready within the ready timeout, as with a missing image tag, the canary stops at Failed. This change points the engine of chat at a missing tag, with 10 minutes for each step to become ready:

chat.yaml
apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat
  namespace: prod
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 4
    maxReplicas: 4
    runner:
      name: ome-container
      image: docker.io/lmsysorg/sglang:v0.0.0-missing
  rollout:
    groups:
      - components:
          - engine
        canary:
          readyTimeout: 10m
          steps:
            - capacity: "25%"
              traffic: 10
              pause: {}
            - capacity: "100%"
              traffic: 100
Shell
kubectl apply -f chat.yaml
Output
inferenceservice.ome.io/chat configured

Step 1's new Instance never becomes ready, so status shows PHASE Pending. After 10 minutes, PHASE shows Failed, and the canary stays there while the Instances on the stable revision keep serving. The ome.io/rollout-ready-timeout annotation overrides readyTimeout, and the ome-resources chart sets a default of 15m for groups that set neither. readyTimeout has the details.

promote refuses on a failed canary. To recover:

  1. Roll back the canary, and wait until PHASE shows RolledBack.
  2. Fix the image in chat.yaml, and apply it again. The new revision starts a fresh canary at step 1, and OME removes the ome.io/rollout-rollback annotation.

When a canary pod keeps crashing

A canary can park Failed long before its ready timeout, with a Warning event CanaryPodCrashed that names a pod. That pod served, then its runner container, ome-container, kept dying. A single death doesn't fail the canary: it's a dip in capacity that the step waits out, while the kubelet restarts the container in place or the Instance restart policy rebuilds the Instance. When the runner dies again after that, at any step since the canary began, OME parks the canary where it stands, without waiting for a gate or the ready timeout — even when no timeout is set. A sidecar or an init container that restarts doesn't park the canary, and neither does a pod that crashes before it first serves: that waits in Pending for the ready timeout, as When a step never becomes ready shows.

Unlike a ready-timeout park, the crash park returns the step's recorded traffic to the stable revision: status records 0% for the new revision. The stable Instances keep serving, and the canary keeps its Instances, the crashing pod among them.

Read the event for the pod and what killed it:

Shell
kubectl get events -n prod --field-selector reason=CanaryPodCrashed -o custom-columns=TYPE:.type,OBJECT:.involvedObject.name,MESSAGE:.message
Output
TYPE      OBJECT   MESSAGE
Warning   chat     engine canary pod chat-engine-3-default-0 keeps dying after it was serving (container ome-container CrashLoopBackOff (exit 1)); the ladder is parked at step 0 and the step's traffic is back on the stable revision

The message names the component, the pod, the termination the kubelet recorded and the step. It counts steps from 0, like the promote preview, so step 0 is the first step, the one STEP shows as 1/2. Read the crashed run's logs with kubectl logs chat-engine-3-default-0 -n prod -c ome-container --previous.

promote refuses on the parked canary. Recover as after a ready timeout: roll back the canary, then fix what crashes the server and apply the fix, whose new revision starts a fresh canary at step 1. Crash-looping canary pods has the exact reading.

Troubleshooting

When promote or rollback refuses, it prints error: and the reason, changes nothing, and exits 1. Refusals lists the messages and their causes.

promote refuses at the gate

error: action refused: promote requires an active indefinite manual gate; analysis requires explicit override means the canary isn't waiting at a gate that this promote can pass. Run status and check:

  • GATE: plain promote needs Manual, and --override-analysis --yes needs Analysis.
  • PHASE: the canary must be Paused, or Promoting at the last step. Pending means the step's new Instances are still starting. Failed means the ready timeout passed before they were ready, or while the step's analysis got only inconclusive results, or a canary pod kept dying after it served: see When a step never becomes ready and When a canary pod keeps crashing.
  • An earlier promote that OME is still finishing: wait, and run status again.
  • HOLD CanaryPreStep in explain, after a repin: promote can't release this hold. Set the promote annotation by hand, as Repin a drifted rollout plan shows.

A promote or rollback request is present

error: action refused: a rollout promote or rollback mailbox is present means the InferenceService has the ome.io/rollout-promote or ome.io/rollout-rollback annotation, whatever its value, even false. A promote request clears once OME advances the step. The rollback annotation stays after a rollback until a new revision starts a fresh canary, as Roll back the canary describes. A promote request that names an earlier canary revision, as when the target changed first, never advances the canary. Remove it:

Shell
kubectl annotate inferenceservice chat -n prod ome.io/rollout-promote-
Output
inferenceservice.ome.io/chat annotated

Other refusals and errors

Message Meaning
action refused: no applicable active rollout or lifecycle work was observed The canary already completed or rolled back, or hasn't started.
action refused: globally paused canary cannot consume a request The rollout is paused. Resume it first.
action refused: controller safety evidence is stale or inconsistent OME is still catching up with a recent change: wait, run status, and retry. More than one canary group always causes it.
guarded annotation patch rejected; refresh rollout status and retry explicitly The InferenceService changed after the command read it. Check status, and run the command again. It exits 3.

If the command fails after it sent the patch, as with a timeout, the request may have been stored. Run status before you run the command again, as When the outcome is unknown explains.

Next steps