Serve a multi-node model Unreleased: coming in v1.3

Run one copy of a model across several nodes by giving its engine a leader and workers, which OMENative creates, updates and repairs together as one Instance.

Run one copy of a model across several nodes, as a leader pod and worker pods that serve it together. OMENative runs each leader and its workers as one Instance, which OME creates, updates, repairs and removes as a unit. By default, an update brings a new Instance up and serving before the old one drains. A leader and a worker on the engine are all it takes to run on OMENative.

You'll serve Qwen3 0.6B, the model from Serve your first model, on two pods with one GPU each, so you can try the steps on modest hardware. A larger model takes the same steps, with more GPUs in each pod or more workers, and often a longer readiness deadline.

Before you begin¶

OME v1.3 or later, and kubectl, with the rights to create namespaces and cluster-scoped OME resources.
The ClusterBaseModel qwen3-0-6b from Serve your first model, downloaded and Ready.
Two amd64 nodes with a free NVIDIA GPU each, exposed as nvidia.com/gpu, and the model Ready on both. kubectl get nodes -l models.ome.io/clusterbasemodel.qwen3-0-6b=Ready lists them. The leader and the worker each request 1 GPU, 10 CPUs and 30 GiB of memory. The scheduler can also place both pods on one node with two free GPUs.
Pod-to-pod network traffic between those nodes.
Recommended: the scheduler-plugins PodGroup CRD and a gang-aware scheduler, such as the alpha OME scheduler, which needs Kubernetes 1.35. With them, the leader and its worker are scheduled together or not at all. Gang scheduling shows the setup. Without them, the example still runs when both GPUs are free.

Step 1: Prepare a multi-node runtime¶

Save this runtime as srt-qwen3-0-6b-multi-node.yaml. It's the srt-qwen3-0-6b runtime from Serve your first model, split into a leader and a worker in its engineConfig:

srt-qwen3-0-6b-multi-node.yaml

apiVersion: ome.io/v1beta1
kind: ClusterServingRuntime
metadata:
  name: srt-qwen3-0-6b-multi-node
spec:
  supportedModelFormats:
    - modelFramework:
        name: transformers
        version: "4.51.0"
      modelFormat:
        name: safetensors
        version: "1.0.0"
      modelArchitecture: Qwen3ForCausalLM
      autoSelect: false
      priority: 1
  protocolVersions:
    - openAI
  modelSizeRange:
    min: 0.5B
    max: 1B
  engineConfig:
    tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
    volumes:
      - name: dshm
        emptyDir:
          medium: Memory
    leader:
      runner:
        name: ome-container
        image: docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64
        ports:
          - containerPort: 8080
            name: http1
            protocol: TCP
        command:
          - python3
          - -m
          - sglang.launch_server
          - --host
          - "0.0.0.0"
          - --port
          - "8080"
          - --model-path
          - $(MODEL_PATH)
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --mem-frac
          - "0.9"
          - --tp-size
          - $(PARALLELISM_SIZE)
          - --dist-init-addr
          - $(OME_LEADER_ADDRESS):5000
          - --nnodes
          - $(OME_INSTANCE_POD_COUNT)
          - --node-rank
          - $(OME_INSTANCE_POD_RANK)
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm
        resources:
          requests:
            cpu: 10
            memory: 30Gi
            nvidia.com/gpu: 1
          limits:
            cpu: 10
            memory: 30Gi
            nvidia.com/gpu: 1
        startupProbe:
          httpGet:
            path: /health_generate
            port: 8080
          initialDelaySeconds: 60
          periodSeconds: 6
          timeoutSeconds: 30
          failureThreshold: 150
        readinessProbe:
          httpGet:
            path: /health_generate
            port: 8080
          periodSeconds: 60
          timeoutSeconds: 200
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 60
          timeoutSeconds: 60
          failureThreshold: 5
    worker:
      size: 1
      runner:
        name: ome-container
        image: docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64
        command:
          - python3
          - -m
          - sglang.launch_server
          - --host
          - "0.0.0.0"
          - --port
          - "8080"
          - --model-path
          - $(MODEL_PATH)
          - --served-model-name
          - Qwen/Qwen3-0.6B
          - --mem-frac
          - "0.9"
          - --tp-size
          - $(PARALLELISM_SIZE)
          - --dist-init-addr
          - $(OME_LEADER_ADDRESS):5000
          - --nnodes
          - $(OME_INSTANCE_POD_COUNT)
          - --node-rank
          - $(OME_INSTANCE_POD_RANK)
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm
        resources:
          requests:
            cpu: 10
            memory: 30Gi
            nvidia.com/gpu: 1
          limits:
            cpu: 10
            memory: 30Gi
            nvidia.com/gpu: 1

leader.runner and worker.runner each define the pod's container, ome-container, and run the same SGLang command. OME sets variables that give each pod its place in the Instance:

Flag	Variable	Value
`--tp-size`	`PARALLELISM_SIZE`	The pod's GPUs times the pods in the Instance: `2` here
`--dist-init-addr`	`OME_LEADER_ADDRESS`, with port 5000	The leader's DNS name, which resolves once the leader pod has an IP address
`--nnodes`	`OME_INSTANCE_POD_COUNT`	`1` + `worker.size`: `2` here
`--node-rank`	`OME_INSTANCE_POD_RANK`	`0` on the leader, `k + 1` on worker `k`

Pod labels and environment lists the other variables, and Serving runtimes explains when OME leaves PARALLELISM_SIZE unset.

tolerations and volumes on engineConfig apply to both pods.
The leader is the pod that serves requests, so only the leader declares the HTTP port and has probes. Keep serving probes off the workers: a failing readiness probe keeps the Instance from becoming ready, and a failing liveness probe can make OME recreate the whole Instance.
worker.size: 1 gives each Instance one worker.
supportedModelFormats and modelSizeRange match those of srt-qwen3-0-6b. With autoSelect: false, OME uses this runtime only when an InferenceService names it.

Create the runtime:

Shell

kubectl apply -f srt-qwen3-0-6b-multi-node.yaml

Output

clusterservingruntime.ome.io/srt-qwen3-0-6b-multi-node created

Step 2: Create the InferenceService¶

Create a namespace for the InferenceService:

Shell

kubectl create namespace qwen3-multi-node

Output

namespace/qwen3-multi-node created

Save this InferenceService as qwen3-multi-node-isvc.yaml:

qwen3-multi-node-isvc.yaml

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-multi-node
  namespace: qwen3-multi-node
spec:
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b-multi-node
  engine:
    minReplicas: 1
    maxReplicas: 1
    leader: {}
    worker:
      size: 1

runtime.name names the runtime from Step 1.
leader: {} and worker make the engine's Service send requests only to leaders, and {} keeps the runtime's leader as it is. Declare both: the webhook rejects an engine that declares only one.
worker.size is the number of workers in each Instance. It defaults to the runtime's worker.size, or 1.
minReplicas and maxReplicas count Instances, and each Instance needs the GPUs of a leader and all its workers. Here, that's one Instance of two pods.
To use a gang-aware scheduler, add its name as schedulerName under engine, as Bring a gang-aware scheduler shows.

Create the InferenceService:

Shell

kubectl apply -f qwen3-multi-node-isvc.yaml

Output

inferenceservice.ome.io/qwen3-multi-node created

Step 3: Check the Instance¶

Wait for the InferenceService to be ready. The first start pulls the SGLang image onto both nodes, which can take several minutes:

Shell

kubectl wait --for=condition=Ready inferenceservice/qwen3-multi-node -n qwen3-multi-node --timeout=30m

Output

inferenceservice.ome.io/qwen3-multi-node condition met

OME created the InferenceReplica qwen3-multi-node-engine, which runs the engine's Instances. Its counts are Instances, not pods:

Shell

kubectl get inferencereplica qwen3-multi-node-engine -n qwen3-multi-node

Output

NAME                      COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
qwen3-multi-node-engine   engine      1         1         1       1           9m

List the Instance's pods with their ranks and nodes:

Shell

kubectl get pods -n qwen3-multi-node -l ome.io/inferenceservice=qwen3-multi-node \
  -o 'custom-columns=NAME:.metadata.name,RANK:.spec.containers[0].env[?(@.name=="OME_INSTANCE_POD_RANK")].value,NODE:.spec.nodeName,READY:.status.conditions[?(@.type=="Ready")].status'

Output

NAME                                 RANK   NODE         READY
qwen3-multi-node-engine-0-leader-0   0      gpu-node-1   True
qwen3-multi-node-engine-0-worker-0   1      gpu-node-2   True

Your node names differ. If the PodGroup CRD is installed, check the Instance's PodGroup:

Shell

kubectl get podgroup qwen3-multi-node-engine-0 -n qwen3-multi-node \
  -o custom-columns=NAME:.metadata.name,MIN-MEMBER:.spec.minMember

Output

NAME                        MIN-MEMBER
qwen3-multi-node-engine-0   2

MIN-MEMBER is the Instance's two pods, so a gang-aware scheduler binds both or neither.

Step 4: Send a request¶

The engine's Service, qwen3-multi-node-engine, sends requests only to the leader. Forward a local port to it, and leave the command running:

Shell

kubectl port-forward svc/qwen3-multi-node-engine 8080:8080 -n qwen3-multi-node

Output

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

With ingress off, the default, don't send requests to the InferenceService's URL. It points to the Service qwen3-multi-node, which also selects the worker. To reach a multi-node engine from outside the cluster, see Expose a multi-node engine.

In a second terminal, send a chat completion request. model is the name that the runtime serves the model under, and chat_template_kwargs turns off Qwen3's thinking mode:

Shell

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is Kubernetes? Answer in one sentence."}], "max_tokens": 100, "chat_template_kwargs": {"enable_thinking": false}}'

The response is an OpenAI chat completion, with the model's answer in choices[0].message.content.

Control how Instances update and recover¶

The engine's lifecycle sets how OME updates and repairs its Instances. This version of qwen3-multi-node-isvc.yaml sets it explicitly and runs two Instances, which need four GPUs:

qwen3-multi-node-isvc.yaml

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-multi-node
  namespace: qwen3-multi-node
spec:
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b-multi-node
  engine:
    minReplicas: 2
    maxReplicas: 2
    leader: {}
    worker:
      size: 1
    lifecycle:
      restartPolicy: RecreateInstanceOnPodRestart
      readyPolicy: AllPodReady
      instanceReadyTimeout: 45m
      updateStrategy:
        type: SurgeThenDrain
        rollingUpdate:
          maxSurge: 1
          maxUnavailable: 1

Field	Default	What it does
`restartPolicy`	`RecreateInstanceOnPodRestart`	When a pod of a ready Instance fails, disappears or restarts its `ome-container`, OME recreates the whole Instance.
`readyPolicy`	`AllPodReady`	An Instance is ready when all its pods are. An engine with a leader and a worker must use this value.
`instanceReadyTimeout`	`30m` in the chart	How long a new Instance has to become ready before OME marks it `Failed`.
`updateStrategy.type`	`SurgeThenDrain`	An update brings up a whole new Instance, then drains an old one once the new one is ready.
`maxSurge`	`25%` of the Instances, rounded up, in the chart	How many new Instances an update brings up at once.
`maxUnavailable`	No limit under `SurgeThenDrain`	How many Instances OME rebuilds at once when a pod is stuck, for example in `CrashLoopBackOff`. See The crash-loop repair.

Apart from instanceReadyTimeout and maxUnavailable, these values are the defaults. maxUnavailable: 1 lets OME rebuild one stuck Instance at a time, which 0 would block. It also keeps the engine Ready while one Instance is out of service. With maxSurge: 1, an update replaces one Instance at a time, so it needs two more GPUs while the new Instance starts. Apply the file:

Shell

kubectl apply -f qwen3-multi-node-isvc.yaml

Output

inferenceservice.ome.io/qwen3-multi-node configured

Instance restart policy and OMENative update strategies explain the other policies and strategies.

Troubleshooting¶

Pods stay Pending¶

kubectl describe pod on a Pending pod shows FailedScheduling events that say what the pod lacks, such as a free GPU or a node where the model is Ready. If the pod has no events at all, check that the scheduler in its spec.schedulerName is running.

If all the Instance's pods stay Pending under a gang-aware scheduler, the whole gang doesn't fit yet. Free GPUs for the leader and all its workers on nodes where the model is Ready. The OME scheduler also needs a topology key, as Bring a gang-aware scheduler explains.

If some pods run while others stay Pending, the running pods keep their GPUs while they wait. There are two causes:

Cause	Sign	Fix
The PodGroup CRD is missing, so OME creates no PodGroup.	The engine's `GangSchedulingUnavailable` condition has the reason `PodGroupCRDNotInstalled`.	Install the CRD and restart the controller, as Install the PodGroup CRD shows.
The pods use the default scheduler, which ignores PodGroups.	A `MaybeNoGangScheduler` Warning event on the InferenceService.	Name a gang-aware scheduler in the engine's `schedulerName` or the runtime's `spec.schedulerName`.

Read the condition's reason:

Shell

kubectl get inferenceservice qwen3-multi-node -n qwen3-multi-node \
  -o jsonpath='{.status.components.engine.lifecycle.conditions[?(@.type=="GangSchedulingUnavailable")].reason}'

Output

PodGroupCRDNotInstalled

Read the event's message:

Shell

kubectl get events -n qwen3-multi-node --field-selector reason=MaybeNoGangScheduler \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

Output

OMENative component=engine created scheduler-plugins PodGroup objects but pod template's spec.schedulerName is "" (default kube-scheduler does not enforce PodGroup gang); set runtime.spec.schedulerName to a gang-aware scheduler (e.g. scheduler-plugins-scheduler) or install scheduler-plugins as a plugin in the default scheduler

If you added the Coscheduling plugin to your default scheduler, the warning is a false positive.

With the chart's settings, a pod that stays unschedulable for 15 minutes makes OME fail the Instance, with an InstanceFailed Warning event that carries the scheduler's message.

Workers can't reach the leader¶

The pods run, but the leader never becomes ready, and the engine's logs show that it can't connect to the other ranks. Read them with kubectl logs -n qwen3-multi-node qwen3-multi-node-engine-0-worker-0, or with the leader's pod name. Check the leader address that the worker connects to:

Shell

kubectl get pod qwen3-multi-node-engine-0-worker-0 -n qwen3-multi-node \
  -o jsonpath='{.spec.containers[0].env[?(@.name=="OME_LEADER_ADDRESS")].value}'

Output

qwen3-multi-node-engine-0-leader-0.qwen3-multi-node-engine-headless

Then check these causes:

The runtime still passes LWS_* variables, which reach the engine as literal text. Make an OMENative copy of the runtime.
The pods run with hostNetwork: true but without dnsPolicy: ClusterFirstWithHostNet, so they use the node's DNS, which can't resolve the leader's name. The catalog's multi-node runtimes set both.
A NetworkPolicy blocks traffic between the pods. Allow it between the pods with the label ome.io/inferenceservice=qwen3-multi-node.

The whole Instance keeps restarting¶

The Instance's pods keep terminating together and coming back. Each time OME recreates the Instance, it records a RestartTriggered Warning event on the InferenceService that names the cause. Read the causes:

Shell

kubectl get events -n qwen3-multi-node --field-selector reason=RestartTriggered \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

For a worker whose container ran out of memory, the output is:

Output

OMENative component=engine instance=0 restart triggered: pod qwen3-multi-node-engine-0-worker-0 container ome-container restarted after Ready: OOMKilled (exit 137) (incarnation=2)

Cause	Fix
`OOMKilled`	The container used more memory than its limit. Raise `resources.limits.memory` in the runtime.
A failed liveness probe	Loosen the probe, or leave probes off the workers, as the runtime in Step 1 does.
`pod count 1 below desired 2`, or `gang member lost: 1 of 2 pods present`	A pod went missing, for example because someone deleted it or its node went away.

Observe a restart shows the other restart events, and how to read an Instance's last failure.

Clean up¶

Delete the InferenceService, its namespace and the runtime:

Shell

kubectl delete inferenceservice qwen3-multi-node -n qwen3-multi-node
kubectl delete namespace qwen3-multi-node
kubectl delete clusterservingruntime srt-qwen3-0-6b-multi-node

Output

inferenceservice.ome.io "qwen3-multi-node" deleted
namespace "qwen3-multi-node" deleted
clusterservingruntime.ome.io "srt-qwen3-0-6b-multi-node" deleted

The ClusterBaseModel qwen3-0-6b stays. Serve your first model shows how to delete it.

Next steps¶

Move from LeaderWorkerSet to OMENative: move an engine that runs as a LeaderWorkerSet onto OMENative.
Use the OME scheduler: place each Instance's leader and workers in one accelerator domain. It's alpha and needs Kubernetes 1.35.
Gang scheduling: schedule the pods of each Instance together.
Set Instance readiness deadlines: decide when a slow or stuck Instance fails.