Deployment modes and OMENative

An InferenceService component, the engine, decoder or router, runs in a deployment mode, which decides what creates and manages its pods: a Kubernetes Deployment, or OMENative. On OMENative, the component's replicas run as Instances. An Instance is one pod, or a leader and its workers, and OME creates, updates, repairs and moves it as one unit. You can put a whole InferenceService on OMENative, or one component.

The deployment modes¶

All three modes put a Service named {isvc}-{component} in front of the component. On RawDeployment, a change is a plain Deployment update, with no rollout group, canary or held revision. A RawDeployment or OMENative component also gets a PodDisruptionBudget, from its minAvailable or maxUnavailable or the ome-resources chart's default of maxUnavailable: 1: see Configure pod disruption budgets. Autoscaler policy explains when it gets an autoscaler.

An InferenceService with both an engine and a decoder is prefill-decode disaggregated, a shape rather than a mode: the engine runs prefill and the decoder runs decode. On OMENative, each gets its own InferenceReplica, and the two can roll out as one rollout group.

Opt in to OMENativeUnreleased: coming in v1.3¶

Opting in gives any component, even one with a single pod per replica, what OMENative does for its Instances:

• By default, an update brings the new pod up and serving before the old one drains.
• With the chart's settings, an update that fails because of the new revision itself, as with an image that can't be pulled, is retried with backoff. After the last attempt, OME holds the revision until a newer revision replaces it or you release it.
• You can migrate an Instance off its node, and the replacement starts before the original is removed.
• Status has a row for each Instance, with its phase, revisions and pod counts.

Rollout groups, canaries included, need OMENative declared in the InferenceService, with spec.deploymentMode or the component's own annotation, or the webhook rejects them with CoordinationRequiresOMENative or CanaryRequiresOMENative. A leader, a worker or the runtime's annotation doesn't count.

For the whole InferenceService¶

Set spec.deploymentMode: OMENative to run all the components on OMENative, the router included. The field takes OMENative or RawDeployment. A component's ome.io/deploymentMode annotation, from the InferenceService or the runtime, wins over it, so you can keep the router on a Deployment with ome.io/deploymentMode: RawDeployment in spec.router.annotations.

This InferenceService serves llama-3-2-3b-instruct from Pre-configured models and runtimes with one engine Instance:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-3b-instruct
  namespace: llama-demo
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-3b-instruct
  runtime:
    name: srt-llama-3-2-3b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

Serve a model on OMENative applies an InferenceService like this one and checks its InferenceReplica, Instance and pod.

OMENative ignores a component's deploymentStrategy and paces rollouts with its lifecycle.updateStrategy instead: see OMENative update strategies. The webhook can warn about the unused field. For the engine under spec.deploymentMode: OMENative, the warning is:

engine.deploymentStrategy is set but deploymentMode is "OMENative"; deploymentStrategy applies only to RawDeployment and is ignored here — use engine.lifecycle.updateStrategy.rollingUpdate for rollout pacing

For one component¶

To choose one component's mode, set its ome.io/deploymentMode annotation. This version annotates the engine and adds a router, which runs as a Deployment because nothing sets its mode:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-3b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-3b-instruct
  runtime:
    name: srt-llama-3-2-3b-instruct
  engine:
    annotations:
      ome.io/deploymentMode: OMENative
    minReplicas: 1
    maxReplicas: 1
  router:
    minReplicas: 1
    maxReplicas: 1

With both an engine and a decoder, declare both OMENative or neither. The webhook compares their annotations in the InferenceService, or spec.deploymentMode, and rejects a mismatch. For an engine annotated OMENative next to a decoder with no mode set, the message is:

InvalidDeploymentModeCombination: engine deploymentMode="OMENative" does not match decoder deploymentMode=""; when either is OMENative, both must use the same value

Change a component's mode¶

When a component's mode changes, OME creates the new workload and leaves the old Deployment, InferenceReplica or LeaderWorkerSet running. The old pods keep their GPUs, so the new pods can stay Pending until you delete the old workload. When the component moves to OMENative, its Service {isvc}-{component} switches to the new pods at once, so it serves nothing until they're ready.

While ingress creation is off, as it is by default, the external Service {isvc} fronts the router, or the engine when there's no router, unless the InferenceService is cluster-local. It sends requests to that component's old and new pods alike until you delete the old workload.

For example, after you move the engine of llama-3-2-3b-instruct from a Deployment to OMENative, delete the engine's Deployment:

kubectl delete deployment llama-3-2-3b-instruct-engine -n llama-demo

deployment.apps "llama-3-2-3b-instruct-engine" deleted from llama-demo namespace

How OME resolves the mode¶

For a component the InferenceService declares, OME merges in the engineConfig, decoderConfig or routerConfig of its serving runtime, and the InferenceService's values win. It takes the component's mode from the first of these that sets one:

1. The component's ome.io/deploymentMode annotation, in the InferenceService or the runtime: spec.engine.annotations or engineConfig.annotations for the engine. OME accepts RawDeployment, OMENative, MultiNode (deprecated) and VirtualDeployment, and skips any other value.
2. The InferenceService's spec.deploymentMode.
3. A leader or worker on the engine or the decoder, from the InferenceService or the runtime, which makes the component OMENative. The router has no leader or worker.
4. RawDeployment.

So a runtime's annotation wins over spec.deploymentMode, unless the InferenceService sets the annotation on the component itself. In the InferenceService's own metadata.annotations, the annotation sets no component's mode: only VirtualDeployment works there, as Other mode values shows.

kubectl ome runtime effective shows the components' modes and the rules that set them, as in RawDeployment (Default).

When OME picks the runtime for you, it skips runtimes whose engineConfig or decoderConfig annotation declares a different mode from the InferenceService's for that component: see Runtime deployment-mode matching.

OMENativeUnreleased: coming in v1.3¶

For an OMENative component, OME writes an InferenceReplica named {isvc}-{component} from the InferenceService, and the InferenceReplica controller creates, updates and replaces the component's Instances. Make your changes in the InferenceService, and get or describe the InferenceReplica, short name irep, to see what's happening.

Instances¶

An OMENative component runs as Instances. An Instance is one pod, or, with a leader and a worker, a leader pod and worker.size worker pods that OME starts, stops and replaces together. The autoscaler, minReplicas and maxReplicas count Instances, not pods.

OME names the pods {isvc}-{component}-{index}-{runner}-{ordinal}, as in llama-3-2-3b-instruct-engine-0-default-0 for the engine above: see What an Instance is.

What OME creates¶

Besides the pods, OME creates these objects for an OMENative component:

OME also creates the PodDisruptionBudget and autoscaler from The deployment modes. It adds a PodMonitor that scrapes all the pods, workers included, when the Prometheus operator's PodMonitor CRD was installed before the manager started.

For a multi-pod Instance, the Service selects only leaders when the InferenceService itself sets worker on the component, and workers too when only the runtime does. To select only leaders, add leader: {} and worker: {} to the component in the InferenceService, which keep the runtime's settings. Without a router, the external Service {isvc} sends requests to the workers too.

Pod labels and environment¶

OME adds these labels to the pods, on top of those in the component's pod template. Where a key is in both, OME's value wins.

It also sets these environment variables:

OME sets them in all containers but the init containers, over any variable with the same name. The pods of a multi-pod Instance that request google.com/tpu also get TPU_WORKER_HOSTNAMES, TPU_WORKER_ID and the other TPU_* variables that multi-host JAX reads. A runtime written for a LeaderWorkerSet uses these instead of the LWS_* variables and labels: Move from LeaderWorkerSet to OMENative has the mapping.

OMENative pods also have the readiness gate ome.io/serving. OME sets it to False to take the pod out of rotation without stopping it, as it does during a drain or before an in-place update.

Observe OMENative in statusUnreleased: coming in v1.3¶

An OMENative component reports conditions, Instance counts and a row per Instance.

Conditions¶

The InferenceReplica's Ready condition says whether the component serves, so you can wait on it:

kubectl wait --for=condition=Ready inferencereplica/llama-3-2-3b-instruct-engine -n llama-demo --timeout=15m

inferencereplica.ome.io/llama-3-2-3b-instruct-engine condition met

A rollout in flight makes the condition Unknown, so kubectl wait keeps waiting until it finishes. The reason is the first row in this table that applies:

The minimum is the desired number of Instances less the maxUnavailable of the component's update strategy, or all of them when it isn't set. Staged means the rollout holds at a partition on purpose: see OMENative update strategies.

Other conditions flag problems without changing Ready:

OME also sets the InferenceService's EngineReady, DecoderReady or RouterReady condition for an OMENative component, with the same minimum:

Lifecycle status¶

The InferenceService reports an OMENative component's Instance counts and revisions under status.components.{component}.lifecycle, a LifecycleStatus. Show the engine's:

kubectl get inferenceservice llama-3-2-3b-instruct -n llama-demo \
  -o 'custom-columns=REPLICAS:.status.components.engine.lifecycle.replicas,READY:.status.components.engine.lifecycle.readyReplicas,SERVING:.status.components.engine.lifecycle.servingReplicas,REVISION:.status.components.engine.lifecycle.currentRevision'

REPLICAS   READY   SERVING   REVISION
1          1       1         llama-3-2-3b-instruct-engine-5d8f9c7a

readyReplicas, servingReplicas and availableReplicas count Instances as Readiness and availability defines them. The other fields:

rolloutHold names one of these gates:

• Plan waits for the rollout group to settle on a valid plan.
• Pairing keeps a compatible engine and decoder pair serving while the pairing protocol changes.
• Ratio keeps the ratio between the components within tolerance.
• Sequential waits for another component's turn or soak.
• Budget keeps to the surge and unavailability budgets.
• RetryBlock backs off before it retries a failed update.
• Held stops updates to a revision whose retries ran out, until a newer revision replaces it or you release it.

Per-Instance status¶

OME keeps a row for each Instance on the InferenceReplica only, in status.instanceStatuses, or in status.instanceStatusColumns in the compact ColumnarV2 encoding. The chart turns that encoding on, and OME then uses it whenever it's smaller. Where Instance status lives explains the fields.

kubectl ome instance reads the rows in either encoding. For an InferenceService named chat in the namespace prod, with one ready engine Instance:

kubectl ome instance list chat -n prod

COMP     IDX/INC   PHASE   PODS    REVS          AOF   EVIDENCE
engine   0/1       Ready   1/1/1   chat...ne-a   A--   OK

MultiNode (deprecated)¶

A MultiNode component runs as a LeaderWorkerSet named lws-{isvc}-{component}, with a leader pod and its worker pods per replica. It needs LeaderWorkerSet installed in the cluster, and gets no PodDisruptionBudget or autoscaler. Its Service {isvc}-{component} sends traffic to the leader pods of every LeaderWorkerSet in the namespace, so keep the component's LeaderWorkerSet the only one there.

Other mode values¶

Next steps¶

• OMENative: what OMENative does for a component.
• Serve a model on OMENative: opt an InferenceService in and check its Instances.
• Serve a multi-node model: run an engine as a leader and workers.
• OMENative update strategies: how OMENative replaces an Instance's pods when the spec changes.
• Move from LeaderWorkerSet to OMENative: move MultiNode components off LeaderWorkerSet.