Rollout groups Unreleased: coming in v1.3

A rollout group rolls a change out to components of an InferenceService as a unit. Put the engine and the decoder in one group, and prefill and decode move to each new revision together, by blue-green, rolling update or canary. A canary moves capacity in steps, and a step can wait for a timer, for you, or for Prometheus analysis that advances the canary or rolls it back.

You declare up to three groups in spec.rollout.groups. A component can be in only one group, and a component in no group rolls out on its own.

Groups need OMENative declared on the InferenceService, with spec.deploymentMode: OMENative or the component's ome.io/deploymentMode annotation. A component that's OMENative only because of its leader and worker, or its runtime, doesn't count, as Opt in to OMENative explains. A single-node component runs as RawDeployment by default, where a change is a plain Deployment update, with no group.

Roll prefill and decode together¶

This InferenceService serves the pre-configured model mistral-7b-instruct-v0-2. Its prefill-decode runtime, srt-mistral-7b-instruct-pd, runs a router, an engine and a decoder, and needs GPU nodes with RDMA NICs, as Serve a prefill-decode model shows. One group holds the engine and the decoder:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: mistral-pd
  namespace: mistral-demo
spec:
  deploymentMode: OMENative
  model:
    name: mistral-7b-instruct-v0-2
  runtime:
    name: srt-mistral-7b-instruct-pd
  engine:
    minReplicas: 1
    maxReplicas: 1
  decoder:
    minReplicas: 1
    maxReplicas: 1
  router:
    minReplicas: 1
    maxReplicas: 1
  rollout:
    groups:
      - components: [engine, decoder]

Apply the file:

kubectl apply -f mistral-pd.yaml

inferenceservice.ome.io/mistral-pd created

• When a change gives both a new revision, they roll out together by blue-green, since the group sets no progression.
• The router is in no group, so it rolls out on its own.
• While a component's update waits, status.components.<component>.lifecycle.rolloutHold names the gate that holds it. Lifecycle status lists the gates.
• An edit to spec.rollout during a rollout applies to the next one.

Each group can set:

Choose a progression¶

A group rolls out blue-green unless it sets another progression.

In a blue-green or rolling-update group, a failed Instance in any component marks the whole group failed. OME records a CoordinationGroupFailed Warning event on the InferenceService, but doesn't stop the other components.

Rolling update¶

maxSurge and maxUnavailable set each component's own budget, as a count or a percentage, 25% by default. With the default SurgeThenDrain update strategy, a component starts a new Instance before it stops an old one, within maxSurge. With any other strategy, it moves within maxUnavailable. An edit that changes one component's pod settings, such as its runner image, must change the others' too, or the webhook rejects it (RollingUpdateLockstepViolation).

This group moves four engines and two decoders one Instance at a time, and keeps their 2:1 ratio:

spec:
  engine:
    minReplicas: 4
    maxReplicas: 4
  decoder:
    minReplicas: 2
    maxReplicas: 2
  rollout:
    groups:
      - components: [engine, decoder]
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 0
        maintainRatio:
          tolerance: 25

• maxSurge: 1 lets each component start one new Instance at a time. Under another strategy, maxUnavailable: 0 would stall it.
• maintainRatio keeps the two in step. OME holds a component while it's more than 25% further through its rollout than the other. It also holds an update that would move their serving ratio more than 25% from 2:1. The held component's rolloutHold gate is Ratio.

tolerance runs from 0 to 100, and 0 allows no drift. When you leave it out, it's 5, the ome-resources value ome.controller.coordination.defaultRatioTolerancePercent. Above 50, the webhook warns that the check is effectively off (RatioToleranceTooHigh). A component whose update strategy is InPlaceIfPossible or InPlaceOnly skips the check.

Canary¶

A canary group moves all its components through one canary's steps. Each step's capacity applies to each component: at "25%", a quarter of the engine's replicas and a quarter of the decoder's, rounded up, run the new revision. The step waits until both are ready. When the InferenceService declares both, the engine and the decoder are one canary unit, so a canary group that names one must name both. The router is a unit of its own, and one canary group can hold both units.

spec:
  rollout:
    groups:
      - components: [engine, decoder]
        canary:
          steps:
            - capacity: "25%"
              traffic: 10
              pause: {}
            - capacity: "100%"
              traffic: 100

The first step waits until you promote it. To promote a step, or to roll the canary back to its stable revision, see Promote or roll back a canary. The router can run a canary of its own at the same time, under groupOrdering: Concurrent. The alpha kubectl ome rollout promote and rollback commands then refuse, since they act only on a rollout with one canary group. Canary progression describes the steps, and Canary metric analysis describes their Prometheus checks.

Change the engine-decoder wire protocol¶

spec.rollout.pairingProtocol is a token you choose, such as kv-v1, for the wire protocol between the engine and the decoder. Change it only when that protocol changes. A new token gives both components a new revision, and OME labels their pods with ome.io/pairing-protocol.

To move from one token to another, roll the engine and the decoder out in one blue-green group. If mistral-pd runs with pairingProtocol: kv-v1, this spec moves it to kv-v2:

spec:
  rollout:
    pairingProtocol: kv-v2
    groups:
      - components: [engine, decoder]
        blueGreen: {}

While the change rolls out, OME keeps at least one engine and decoder pair with the same token serving. Under the default SurgeThenDrain update strategy, each component starts and serves its new-token pods before its old pods drain, and an old pod's drain waits, with the rolloutHold gate Pairing, in two cases:

• The component's own new pods can't pair yet, because no serving pod of the other component runs the new token, and the other component also updates by SurgeThenDrain. Retiring an old pod that still pairs would trade serving capacity for pods that can't pair, so the drain waits, with a reason such as pairing protocol transition to "kv-v2": the replacement engine cannot pair because no serving decoder runs cohort "kv-v2" yet; holding the drain of the serving engine of cohort "kv-v1" until one does. The other component's first pod to serve the new token releases the hold.
• The drain would take the last such pair out of service. It waits until the new token serves on both components.

The holds can't deadlock, because a SurgeThenDrain component's new pods come up whether or not any drain waits. With that strategy on both components, even one engine and one decoder keep a pair serving through the change. Under RecreatePod or an in-place strategy, a component runs the new token only once its old pods leave rotation, and with one engine and one decoder the change can briefly leave no pair.

The webhook rejects a change from one token to another when the engine and the decoder roll out separately or by rolling update (PairingProtocolChangeUncoordinated). It always accepts setting a token for the first time, or clearing it.

Roll components one at a time¶

When the groups that aren't canaries are two or more blue-green groups of one component each, they form a sequence. OME rolls a sequence out one group at a time, in list order. With this spec.rollout, mistral-pd rolls out the decoder first, then the engine:

spec:
  rollout:
    groups:
      - components: [decoder]
        soak: 10m
      - components: [engine]

When a change gives both a new revision, OME:

1. Rolls out the decoder, while the engine waits.
2. Waits out the 10-minute soak once the decoder finishes.
3. Rolls out the engine.

OME skips a component that the change leaves alone. Each wait lasts the longest soak of any group but the last, counted from when the component before it last finished a rollout.

While the decoder rolls out, the engine's rolloutHold shows why it waits:

kubectl get inferenceservice mistral-pd -n mistral-demo -o custom-columns='GATE:.status.components.engine.lifecycle.rolloutHold.gate,REASON:.status.components.engine.lifecycle.rolloutHold.reason'

GATE         REASON
Sequential   Sequential waiting on decoder

During the soak, the reason is Sequential.Soak: engine waiting out the 10m0s soak after decoder.

If a component's rollout fails, the components after it wait, and OME records a CoordinationGroupFailed Warning event. The sequence goes on once another change, such as a fixed image or the previous one, makes the failed component healthy.

How groups run together¶

Each canary group rolls out on its own, alongside the other groups. The other groups roll out at the same time, unless they form a sequence.

spec.rollout.groupOrdering changes only what the webhook accepts, not how the groups roll out. Sequential, the default, accepts one group or a sequence alone. Any other list of two or more groups, such as a canary group beside another group, needs groupOrdering: Concurrent, or the webhook rejects it with GroupOrderingNotHonored.

What admission rejects¶

When the webhook rejects a rollout, kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, then a message that ends in a reason in parentheses:

Canary progression lists the rules for canary groups.

Next steps¶

• Promote or roll back a canary: move a canary past a step that waits for you, or return it to its stable revision.
• kubectl ome rollout: see where a rollout is, and the gate a canary step waits at.
• Canary progression: every field of a canary step, and how a step advances.
• Canary metric analysis: gate a step on Prometheus metrics.
• Pause and resume a rollout: hold a rollout in progress, and let it go on.
• Rollout policy: define a progression once, for groups to share. It's alpha, and off by default.
• Serve a prefill-decode model: run the engine and the decoder on OMENative.