Canary progression Unreleased: coming in v1.3

A canary moves a new revision onto a share of a component's Instances at a time, and waits at a gate after each step. The gate opens at once, after a timed pause, when you promote the step, or when Prometheus analysis passes. A rollback, from you or from a failed analysis, returns every Instance to the stable revision. You declare a canary in the canary block of a rollout group in an InferenceService. To act on a running canary, see Promote or roll back a canary.

A canary group's components must run on OMENative. Set spec.deploymentMode: OMENative, or ome.io/deploymentMode: OMENative in the component's annotations. A multi-node component needs this too, even though it runs on OMENative by default.

Example¶

The InferenceService chat rolls its engine out in three steps:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat
  namespace: prod
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 4
    maxReplicas: 4
  rollout:
    groups:
      - components: [engine]
        canary:
          scaleDownDelaySeconds: 60
          readyTimeout: 20m
          steps:
            - capacity: "25%"
              traffic: 10
              pause:
                duration: 10m
            - capacity: "50%"
              traffic: 50
              pause: {}
            - capacity: "100%"
              traffic: 100

Each step first waits for its new Instances to be ready, and the canary fails if a step waits longer than 20 minutes. kubectl ome rollout status shows the canary's step, gate and phase.

A canary runs when a change, such as a new image, gives the engine a new revision while an earlier one serves. So the first revision of chat rolls out without a canary, and adding the group starts nothing until the next change. The canary keeps the plan it started with: an edit to the canary block waits for the next run. The alpha kubectl ome rollout repin applies it to the running canary, as Repin a drifted rollout plan shows.

Canary prefill and decode together¶

When an InferenceService declares both an engine and a decoder, a canary group that names one must name both, and the two roll out as one canary. The InferenceService chat-pd uses the model and the prefill-decode runtime that Serve a prefill-decode model creates, on GPU nodes with RDMA NICs:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat-pd
  namespace: prod
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: srt-llama-3-2-1b-instruct-pd
  engine:
    minReplicas: 4
    maxReplicas: 4
  decoder:
    minReplicas: 2
    maxReplicas: 2
  router:
    minReplicas: 1
    maxReplicas: 1
  rollout:
    groups:
      - components: [engine, decoder]
        canary:
          steps:
            - capacity: "50%"
              traffic: 50
              pause: {}
            - capacity: "100%"
              traffic: 100

Each step's capacity applies to each component. Step 1 moves 2 of the 4 engine Instances and 1 of the 2 decoder Instances to the new revision. It waits for all three to be ready, then for a promote. The router is in no group, so it rolls out on its own and balances requests across both revisions.

Fields¶

New-revision capacity¶

capacity takes one of these forms:

A component's desired replicas are its minReplicas. Set minReplicas on each component in the group, in the InferenceService itself. Without it, a step can wait for Instances that never start until its ready timeout, or start its gate before any new Instance is ready. Set replica defaults explains why defaults from the runtime or the chart don't count.

A step replaces Instances; it doesn't add any. The rest of the desired replicas stay on the previous revision, and Instances that an autoscaler adds above them run the new revision. Each component works out its own count, as the prefill-decode example shows. For a multi-node component, capacity counts whole Instances, each a leader and its workers.

How a step advances¶

A step starts by moving its capacity to the new revision, and waits in Pending until every component in the group has that many new Instances ready. At that moment, OME records the step's traffic, and the step's gate decides when the next step starts:

GATE in kubectl ome rollout status shows these names. If a step's ready Instances drop below its count, the step goes back to Pending, and its clock starts again once its new Instances are ready. The one capacity loss a step doesn't wait out is a crash-looping canary pod: it parks the canary at Failed before OME checks any gate. At the last step, the gate holds the canary in Promoting until it passes.

Phases¶

The canary's phase is in status.components.<component>.rolloutPhase of the group's primary component: the router when the group includes it, otherwise the engine. PHASE in kubectl ome rollout status shows it:

While a canary runs¶

Traffic weight in status¶

When a step's new Instances are ready, OME records the step's traffic as the new revision's weight and the rest as the stable revision's. It writes both in status.components.<component>.traffic of the primary component:

kubectl ome rollout status shows the weight in TRAFFIC, as target% -> observed%, and explain shows the canary's revisions: see kubectl ome rollout. At step 1 of the example, status records 10% for the new revision, but it has a quarter of the ready Instances and gets about a quarter of the requests.

readyTimeout¶

The ready timeout bounds how long a step waits in Pending for its new Instances, and how long an analysis step goes without a conclusive result. The first of these that's set applies:

With none set, a step never times out: it waits in Pending until its Instances are ready. A crash-looping canary pod still parks the canary at Failed, timeout or none. The timeout counts from when the step last entered Pending. For an analysis, it counts from the last conclusive sample, or from when the step's new Instances were ready if there's none yet.

When it expires, the phase is Failed. The canary keeps its Instances and its recorded traffic, the stable Instances keep serving, and promote refuses. Apply a fix that gives the group a revision other than the failed canary's and the stable one, and a fresh canary starts at step 1 toward it, with the same stable revision. A rollback also works on a failed canary: every Instance returns to the stable revision, as When a step never becomes ready shows. A crash-looping canary pod parks the canary the same way, except that the step's recorded traffic returns to the stable revision at once.

Crash-looping canary pods¶

A canary pod that keeps dying after its Instance served parks the canary where it stands. OME reads the crash before the step's capacity, its ready timeout and its gate, so the park happens at any step, with or without a ready timeout set:

• The phase goes to Failed at the current step, and status.components.<component>.canary.failed.reason records CapacityTimeout, the same reason as a ready-timeout park. The event tells the two apart.
• The step's recorded traffic returns to the stable revision: status records 0% for the new revision, where a ready-timeout park keeps the step's weight.
• OME emits the Warning event CanaryPodCrashed on the InferenceService, naming the component, the pod, its termination detail and the step, counted from 0. In a group of several components, a crash in any member's canary pod parks the group's canary.

The canary keeps its Instances, and the stable Instances keep serving. Recovery is the same as for any Failed canary: a rollback returns every Instance to the stable revision, and a fix that gives the group a revision other than the parked canary's and the stable one starts a fresh canary at step 1.

OME reads a crash loop only from the pods of the new revision, and only from their runner container, ome-container: a stable pod's crash, or a sidecar or init container that restarts, parks nothing. The pod's Instance must have served first:

When a canary pod keeps crashing shows the event and the recovery.

What admission rejects¶

The webhook rejects a canary with a message that names the field and gives one of these reasons in parentheses:

The API server itself rejects more than 20 steps, or a traffic outside 0 to 100. Rollout groups lists the rules for every group.

Related pages¶

• Canary metric analysis: the metric checks an analysis step runs.
• Promote or roll back a canary: act on a canary at its gate.
• Pause and resume a rollout: hold a rollout, canary included.
• Serve a prefill-decode model: run the engine and decoder that the prefill-decode example rolls out.
• Rollout groups: how groups divide a rollout between components.
• Deployment modes: how to opt in to OMENative.
• kubectl ome rollout: the commands that show and act on a canary.
• GroupCanary in the API reference.