Spread Instances across fault domains Unreleased: coming in v1.3

Spread an OMENative engine's or decoder's Instances across racks, fabric slices or zones with topologySpread and topologySpreadKey, as a preference or a requirement.

Spread the Instances of an engine or decoder across zones, racks or fabric slices, so that when one domain fails, the Instances in the others keep serving. Set topologySpread on a component that runs on OMENative, and OME asks the scheduler to spread its Instances across the values of a node label. Here you spread the two Instances of a Qwen3-0.6B engine across zones, then check where they landed.

How the spread works¶

OME adds a Kubernetes topology spread constraint to each Instance's leader pod, or to its only pod, so the scheduler counts the component's Instances per domain. Set these fields on the engine or the decoder:

Field	What it sets	When unset
`topologySpread`	`Required` or `Preferred`.	No spread.
`topologySpreadKey`	The node label whose values are the domains, such as `topology.kubernetes.io/zone`.	OME uses `topologyKey`. With neither key, there's no spread.
`topologyKey`	The node label that keeps a multi-pod Instance's workers in their leader's domain.	Nothing keeps the workers in their leader's domain.

A runtime can set the same fields in its engineConfig or decoderConfig, as defaults that the InferenceService's values override. The engine and the decoder spread their Instances separately. Other deployment modes ignore the fields, and the webhook doesn't warn.

Required and Preferred¶

`topologySpread`	`whenUnsatisfiable`	Where a new Instance goes	Pick it when
`Required`	`DoNotSchedule`	Only to a domain with the fewest of the component's Instances, so counts differ by at most one. When those are full, the pod stays `Pending`, even if another domain has room.	An Instance should wait for room rather than break the spread.
`Preferred`	`ScheduleAnyway`	Preferably to a domain with the fewest Instances, and to another one when those are full.	An Instance should start even if it breaks the spread.

The scheduler counts only the domains that have a node the pods select. The example engine's pods select nodes where the model is Ready, so if the model is Ready in a single zone, Required puts all the Instances there.

Choosing the key¶

Set the keys by the Instance's shape and the domain you expect to fail:

Instance	Domain that fails	Keys to set
One pod	Any	`topologySpreadKey`
A leader and workers	The one that holds the Instance together, such as an NVLink rack	`topologyKey` only
A leader and workers	A larger one, such as a zone of racks	`topologyKey` to the rack label, and `topologySpreadKey` to the zone label

For an engine with a leader and workers, these fields keep each Instance's pods in one NVLink domain and spread the Instances across zones:

YAML

engine:
  topologyKey: nvidia.com/gpu.clique
  topologySpread: Required
  topologySpreadKey: topology.kubernetes.io/zone

Step 1: Set the spread on an InferenceService¶

Create a namespace for the InferenceService:

Shell

kubectl create namespace spread-demo

Output

namespace/spread-demo created

Save this InferenceService as qwen3-0-6b-isvc.yaml:

qwen3-0-6b-isvc.yaml

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: spread-demo
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 2
    maxReplicas: 2
    topologySpread: Required
    topologySpreadKey: topology.kubernetes.io/zone

deploymentMode: OMENative runs the engine on OMENative. A single-pod engine runs as a Deployment by default.
minReplicas and maxReplicas keep the engine at two Instances, each a single pod.
topologySpread: Required keeps the two Instances in different zones.
topologySpreadKey names the zone label.

Apply the file:

Shell

kubectl apply -f qwen3-0-6b-isvc.yaml

Output

inferenceservice.ome.io/qwen3-0-6b created

Wait for the InferenceService to be ready. The first start pulls the SGLang image, which can take several minutes:

Shell

kubectl wait --for=condition=Ready inferenceservice/qwen3-0-6b -n spread-demo --timeout=30m

Output

inferenceservice.ome.io/qwen3-0-6b condition met

Step 2: Check where the Instances landed¶

The engine's InferenceReplica, qwen3-0-6b-engine, shows the spread settings in effect, with the runtime's defaults merged in:

Shell

kubectl get inferencereplica qwen3-0-6b-engine -n spread-demo \
  -o 'custom-columns=NAME:.metadata.name,SPREAD:.spec.topologySpread,SPREAD-KEY:.spec.topologySpreadKey,TOPOLOGY-KEY:.spec.topologyKey'

Output

NAME                SPREAD     SPREAD-KEY                    TOPOLOGY-KEY
qwen3-0-6b-engine   Required   topology.kubernetes.io/zone   <none>

List the engine's pods with their spread constraint and their node:

Shell

kubectl get pods -n spread-demo -l ome.io/inferenceservice=qwen3-0-6b \
  -o 'custom-columns=NAME:.metadata.name,SPREAD-KEY:.spec.topologySpreadConstraints[*].topologyKey,WHEN:.spec.topologySpreadConstraints[*].whenUnsatisfiable,NODE:.spec.nodeName'

Output

NAME                            SPREAD-KEY                    WHEN            NODE
qwen3-0-6b-engine-0-default-0   topology.kubernetes.io/zone   DoNotSchedule   gpu-node-1
qwen3-0-6b-engine-1-default-0   topology.kubernetes.io/zone   DoNotSchedule   gpu-node-2

Each pod is one Instance. On a component with workers, only the leaders carry the constraint, so the workers show <none>.

Show the nodes' zones, using the names from your NODE column:

Shell

kubectl get nodes gpu-node-1 gpu-node-2 \
  -o 'custom-columns=NAME:.metadata.name,ZONE:.metadata.labels.topology\.kubernetes\.io/zone'

Output

NAME         ZONE
gpu-node-1   zone-a
gpu-node-2   zone-b

The two Instances run in different zones, so losing one zone leaves one Instance serving.

Add the spread to a running InferenceService¶

Add the fields to the engine or decoder of an InferenceService that runs on OMENative, and apply it. Running Instances stay where they are. New pods get the constraint: those of a scale-up, a repair, a migration or a rollout. To move a running Instance now, request an Instance migration.

Changing topologyKey on a component with workers is different: OME rolls the component onto a new revision.

Spread under the OME scheduler¶

The OME scheduler is an optional second scheduler that places each opted-in OMENative gang in one accelerator domain, all or nothing. It's alpha, and runs only on Kubernetes 1.35. Use Required with it: when the domain it picks for a gang would break the spread, it tries another. With the scheduler chart's defaults, Preferred has no effect.

Troubleshooting¶

An Instance stays Pending¶

Run kubectl describe pod qwen3-0-6b-engine-1-default-0 -n spread-demo. When the spread holds the pod, its FailedScheduling event counts nodes that didn't match pod topology spread constraints:

Case	Why the pod waits	What to do
The count ends in `(missing required label)`	Those nodes lack the key's label, so `Required` skips them.	Label the nodes, or correct `topologySpreadKey`.
The count has no suffix	The domains with the fewest of the component's Instances are full.	Free room in one of them, or run fewer Instances.
The pod is new in a migration, or in a rollout with the default `SurgeThenDrain` strategy	The old pod counts, and keeps serving, until the new one is ready.	Free room in a domain with the fewest Instances, counting the old pod. See SurgeThenDrain.
Every node in one domain is cordoned, or has a taint the pods don't tolerate	That domain still counts, and its low count holds new Instances back from the others.	The pod schedules once those nodes accept pods again.

The Instances landed in one domain¶

Run the commands in Step 2, and find what you see:

What Step 2 shows	Cause	What to do
`SPREAD` is `<none>`, or the component has no InferenceReplica	The component doesn't run on OMENative, or nothing sets `topologySpread`.	Turn on OMENative for the component, and set `topologySpread`.
`SPREAD-KEY` and `TOPOLOGY-KEY` are both `<none>`	OME has no key to spread on.	Set `topologySpreadKey`.
The pods have no `SPREAD-KEY`, or a `SPREAD-KEY` or `WHEN` from before your change	The pods are older than the change.	Request an Instance migration to move them.
`WHEN` is `ScheduleAnyway`	`Preferred` used another domain while the ones with the fewest Instances were full. Under the OME scheduler's defaults, it has no effect.	Use `Required`.
`WHEN` isn't what you set, on a pod created after your change	Your runtime or InferenceService already sets a spread constraint on that key, and OME keeps it.	Remove that constraint.
All the values are right	Only one domain has a node the pods select, such as one where the model is `Ready`.	Put the model on nodes in a second domain, as Choose the nodes shows.

Clean up¶

Delete the InferenceService and its namespace:

Shell

kubectl delete inferenceservice qwen3-0-6b -n spread-demo
kubectl delete namespace spread-demo

Output

inferenceservice.ome.io "qwen3-0-6b" deleted from spread-demo namespace
namespace "spread-demo" deleted

The model and the runtime stay. Serve your first model shows how to delete them.

Next steps¶

Request an instance migration: move a running Instance, whose new pods get the spread.
Serve a multi-node model: run Instances with a leader and workers.
Use the OME scheduler: install the alpha gang scheduler, and opt a workload in.