Gang scheduling Unreleased: coming in v1.3

With the scheduler-plugins PodGroup CRD installed, OME creates a PodGroup for each multi-pod OMENative Instance, so a gang-aware scheduler places its leader and workers together or not at all.

Gang scheduling places all the pods of a multi-pod Instance together, or none of them. An OMENative Instance with a leader and workers serves only when all its pods run, and gang scheduling stops some of them from holding GPUs while the rest wait.

When the scheduler-plugins PodGroup CRD is installed, OME creates a PodGroup for each multi-pod Instance of an OMENative engine or decoder, and none for RawDeployment or MultiNode (deprecated) components. A gang-aware scheduler reads the PodGroups: the alpha OME scheduler or the scheduler-plugins Coscheduling plugin. The default Kubernetes scheduler ignores them and places pods one at a time.

Install the PodGroup CRD¶

No OME chart installs the PodGroup CRD. Install it from scheduler-plugins v0.34.7, the release OME is built against. If you run the OME scheduler, install the CRD version from Use the OME scheduler instead. Then restart the controller, which looks for the CRD only when it starts. This example assumes OME runs in the ome namespace:

Shell

kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/scheduler-plugins/v0.34.7/config/crd/bases/scheduling.x-k8s.io_podgroups.yaml
kubectl rollout restart deployment ome-controller-manager -n ome

Output

customresourcedefinition.apiextensions.k8s.io/podgroups.scheduling.x-k8s.io created
deployment.apps/ome-controller-manager restarted

Bring a gang-aware scheduler¶

A PodGroup holds a gang together only when the scheduler that places the pods reads it. The pods use the scheduler that the runtime or the InferenceService names, and the default scheduler otherwise. The default scheduler can bind part of an Instance while the rest waits, and the bound pods hold their GPUs with nothing serving. Use one of these:

Scheduler	What it does	`schedulerName`
OME scheduler	Places each gang in one topology domain, all or nothing. Alpha, and needs Kubernetes 1.35.	`ome-scheduler`
scheduler-plugins Coscheduling	Places each gang all or nothing, in your default scheduler or as a second scheduler.	Unset in your default scheduler, or `scheduler-plugins-scheduler` from the scheduler-plugins Helm chart

The OME scheduler also needs a topology key: the component's topologyKey, or its chart's scheduler.plugin.topologyKey for every gang. Without either, the gang's pods stay Pending. Use the OME scheduler installs the scheduler and sets the key.

Set spec.schedulerName in the runtime to use one scheduler for all its components:

YAML

spec:
  schedulerName: ome-scheduler

A schedulerName in an InferenceService component, or in a runtime component config such as engineConfig, overrides it for that component's leader and workers:

YAML

spec:
  engine:
    schedulerName: ome-scheduler

When the pods of a component with PodGroups use the default scheduler, OME records a MaybeNoGangScheduler Warning event on the InferenceService. If you added Coscheduling to your default scheduler, ignore it.

One PodGroup per multi-pod Instance¶

OME creates an Instance's PodGroup before its pods, and names it <isvc>-<component>-<index>. Its minMember is the Instance's pod count: the leader plus worker.size workers. Its scheduleTimeoutSeconds comes from the gang schedule timeout. When the component sets topologyKey, the PodGroup carries it in the ome.io/topology-key annotation. OME labels the Instance's pods with scheduling.x-k8s.io/pod-group, set to the PodGroup's name, which gang-aware schedulers read.

For example, the InferenceService in Serve a multi-node model runs one engine Instance of a leader and one worker, so OME creates the PodGroup qwen3-multi-node-engine-0. Without its owner reference and status, it looks like this:

YAML

apiVersion: scheduling.x-k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: qwen3-multi-node-engine-0
  namespace: qwen3-multi-node
  labels:
    component: engine
    ome.io/inferenceservice: qwen3-multi-node
    ome.io/instance-index: "0"
    ome.io/managed-by: OMENative
spec:
  minMember: 2
  scheduleTimeoutSeconds: 600

List the InferenceService's PodGroups:

Shell

kubectl get podgroups -n qwen3-multi-node -l ome.io/inferenceservice=qwen3-multi-node \
  -o custom-columns=NAME:.metadata.name,MIN-MEMBER:.spec.minMember,TIMEOUT:.spec.scheduleTimeoutSeconds

Output

NAME                        MIN-MEMBER   TIMEOUT
qwen3-multi-node-engine-0   2            600

List the Instance's pods by their PodGroup label:

Shell

kubectl get pods -n qwen3-multi-node -l scheduling.x-k8s.io/pod-group=qwen3-multi-node-engine-0 -o name

Output

pod/qwen3-multi-node-engine-0-leader-0
pod/qwen3-multi-node-engine-0-worker-0

Bound the gang schedule timeout¶

A gang-aware scheduler holds a gang's placed pods until the whole gang fits, for up to the PodGroup's scheduleTimeoutSeconds. At the timeout, it frees their resources for other pods and tries the gang again later.

OME sets scheduleTimeoutSeconds to the component's ready timeout, instanceReadyTimeout, clamped between the min and max of the controller's gangScheduleTimeout setting. The ome-resources chart sets both:

YAML

ome:
  controller:
    lifecycle:
      instanceReadyTimeout: 30m
      gangScheduleTimeout:
        min: 60s
        max: 600s

With these values, the 30-minute ready timeout becomes 600 seconds, as in the PodGroup above.

Ready timeout	`gangScheduleTimeout`	`scheduleTimeoutSeconds`
Between `min` and `max`	Set	The ready timeout
Above `max`	Set	`max`
Below `min`, or unset	Set	`min`
Set	Unset	The ready timeout: 1800 in the kustomize install
Unset	Unset	Unset: the scheduler's default applies

The scheduler's default is defaultPermitTimeoutSeconds for the OME scheduler, 600 in its chart, or permitWaitingTimeSeconds for Coscheduling. gangScheduleTimeout needs both min and max, positive, with min no larger than max. OME treats an invalid one as unset.

Change the cluster defaults shows how to change these settings. OME updates existing PodGroups to the new timeout.

What you see¶

What you see	What it means
The component's `GangSchedulingUnavailable` condition is `True`, with the reason `PodGroupCRDNotInstalled`	The CRD is missing, or was when the controller started. The default scheduler may place part of an Instance, and the OME scheduler keeps its pods `Pending`. Install the CRD.
An `InstanceFailed` Warning event with the scheduler's message	A pod stayed unschedulable longer than the controller's `unschedulableGracePeriod`, 15 minutes with the chart.
An `InstanceFailed` Warning event that names `PodGroupOwnershipConflict`	Another owner controls a PodGroup with the Instance's PodGroup name. OME fails the Instance and creates none of its pods.
A `PodGroupReset` Warning event on the InferenceService	A pod of a running gang failed, so the scheduler-plugins controller marked the PodGroup `Failed`. OME replaced it without failing the Instance. The Instance's restart policy decides what happens next.

The condition is on the component's InferenceReplica, and in the InferenceService's status.components.<component>.lifecycle.conditions. When it's False, its reason is GangSchedulingAvailable. Read its reason for the engine, here without the CRD:

Shell

kubectl get inferenceservice qwen3-multi-node -n qwen3-multi-node \
  -o jsonpath='{.status.components.engine.lifecycle.conditions[?(@.type=="GangSchedulingUnavailable")].reason}'

Output

PodGroupCRDNotInstalled

Next steps¶

The OME scheduler: how OME's alpha scheduler places each gang in one topology domain.
Use the OME scheduler: install the OME scheduler and point your pods at it.
Serve a multi-node model: run an engine whose Instances are a leader and workers.
Set Instance readiness deadlines: the ready timeout that the gang schedule timeout comes from.