OME scheduler configuration Unreleased: coming in v1.3

The ome-scheduler Helm chart's values, the PodGroup fields that the scheduler reads, and its events and metrics.

The ome-scheduler Helm chart installs the OME scheduler as a second scheduler next to your default one, and its values configure it. Pods opt in by setting spec.schedulerName to the scheduler's name. The OME scheduler is alpha, and it runs only on Kubernetes 1.35. Use the OME scheduler installs the chart and opts a workload in.

Chart values¶

Set these values in the values file that you pass to helm install or helm upgrade, as Use the OME scheduler shows. Helm rejects values outside the limits in the Type column, and the combinations in Invalid values. It ignores unknown keys without a warning, so a misspelled key has no effect. {name} stands for the value of scheduler.name.

Name and image¶

Value	Type	Default	What it sets
`scheduler.name`	string: a DNS label	`ome-scheduler`	The name that pods set in `spec.schedulerName`. It also names the Deployment, the leader-election Lease and the chart's other objects.
`global.hub`	string	`""`	A registry that the chart puts in front of `scheduler.image.repository` when the repository has no `/`.
`scheduler.image.repository`	string	`ome-scheduler`	The scheduler's image. OME's releases don't publish it, so build and push it yourself, as Use the OME scheduler shows.
`scheduler.image.tag`	string	`latest`	The image's tag.
`scheduler.image.pullPolicy`	string	`IfNotPresent`	The container's `imagePullPolicy`.
`scheduler.imagePullSecrets`	list	`[]`	The pods' image pull secrets, each with a `name`. When it's empty, `global.imagePullSecrets` applies.
`global.imagePullSecrets`	list	`[]`	The fallback for `scheduler.imagePullSecrets`.

Replicas and availability¶

Value	Type	Default	What it sets
`scheduler.replicaCount`	integer, at least 1	`2`	The number of replicas. Only the leader schedules, and the others wait to take over. More than one replica needs `scheduler.leaderElect: true`.
`scheduler.leaderElect`	boolean	`true`	Leader election between the replicas, through the Lease `{name}` in the release's namespace.
`scheduler.deploymentStrategy.type`	string: `RollingUpdate`	`RollingUpdate`	The Deployment's update strategy.
`scheduler.deploymentStrategy.rollingUpdate.maxUnavailable`	integer of at least 0, or string	`1`	How many replicas an update can stop at once.
`scheduler.deploymentStrategy.rollingUpdate.maxSurge`	integer of at least 0, or string	`0`	How many extra replicas an update can start. With `0`, an update stops a replica before it starts the replacement, so it needs no spare node.
`scheduler.podDisruptionBudget.enabled`	boolean	`true`	Creates the PodDisruptionBudget `{name}` for the scheduler's pods.
`scheduler.podDisruptionBudget.minAvailable`	integer, at least 0	`1`	How many replicas a voluntary disruption, like a node drain, must leave running. It must be below `scheduler.replicaCount`.

Gangs¶

Each scheduler.plugin.<name> value sets the OMEGangPack plugin's <name> argument, and startup errors name the argument. The plugin pins each gang to the domain that it picks, as How it places a gang explains.

Value	Type	Default	What it sets
`scheduler.plugin.topologyKey`	string	`""` (none)	The node label whose values define accelerator domains, for gangs whose PodGroup has no `ome.io/topology-key` annotation, and for standalone packing.
`scheduler.plugin.defaultPermitTimeoutSeconds`	integer, at least 1	`600`	How long a gang's members wait at the scheduler's Permit stage for the rest of the gang, when the PodGroup sets no `scheduleTimeoutSeconds`.
`scheduler.omeControllerIntegration.enabled`	boolean	`true`	Locks `scheduler.plugin.podGroupTopologyKeyAnnotation` to `ome.io/topology-key`, the annotation that OME writes. Turn it off only when another tool creates your PodGroups with a different annotation.
`scheduler.plugin.podGroupTopologyKeyAnnotation`	string, not empty	`ome.io/topology-key`	The PodGroup annotation that names a gang's topology label.
`scheduler.plugin.unsupportedPlacementGroupLabel`	string	`ome.io/placement-group`	The name of the label that keeps a gang `Pending`, as PodGroups and labels describes. An empty value turns this check off.
`scheduler.plugin.podGroupSyncTimeoutSeconds`	integer, at least 1	`30`	How long the scheduler waits at startup to load PodGroups. Without the PodGroup CRD, it starts after the wait, and picks up the CRD when you install it.
`scheduler.plugin.gcIntervalSeconds`	integer, at least 1	`60`	How often the plugin releases the pins of gangs whose pods all finished or were deleted, or whose PodGroup was deleted or re-created.

Packing and preemption¶

Value	Type	Default	What it sets
`scheduler.plugin.standaloneDomainPacking`	boolean	`true`	Steers standalone pods, those without a gang, toward partly used domains, so whole domains stay free for gangs. It needs `scheduler.plugin.topologyKey`, and assumes that all domains have the same number of nodes.
`scheduler.percentageOfNodesToScore`	integer, 1 to 100	Unset	The percentage of nodes that must pass filtering before the scheduler stops looking for more. Unset keeps kube-scheduler's default: all nodes in a cluster of fewer than 100, and a smaller share in larger ones. For standalone packing, set `100` so that its score compares every domain.
`scheduler.omeGangPack.scoreWeight`	integer, at least 1	`10`	The weight of OMEGangPack's score, which does the standalone packing. Keep it at or above `scheduler.nodeResourcesFit.scoreWeight`.
`scheduler.nodeResourcesFit.scoreWeight`	integer, at least 1	`10`	The weight of the `NodeResourcesFit` score, which favors busy nodes.
`scheduler.nodeResourcesFit.resources`	list	`cpu` and `memory`, weight `1` each	The resources that the `NodeResourcesFit` score weighs, each with a `name` and a `weight` of 1 to 100. A larger weight stops the scheduler from starting. Pack by accelerator adds a GPU resource.
`scheduler.disablePodTopologySpreadScore`	boolean	`true`	Turns off the `PodTopologySpread` score, whose spreading works against packing. Topology spread constraints with `whenUnsatisfiable: DoNotSchedule` still apply.
`scheduler.disablePreemption`	boolean	`true`	Turns off kube-scheduler's `DefaultPreemption` plugin for this scheduler, so a gang that doesn't fit waits for capacity instead of evicting pods. The default scheduler still preempts for its own pods.

Metrics endpoint¶

Value	Type	Default	What it sets
`scheduler.metrics.bindAddress`	string, not empty	`0.0.0.0`	The address of the scheduler's HTTPS server, passed as `--bind-address`.
`scheduler.metrics.securePort`	integer, 1 to 65535	`10259`	The HTTPS port for `/metrics` and the health checks, passed as `--secure-port`. The container names it `https-metrics`.
`scheduler.metrics.authDelegation.enabled`	boolean	`true`	Binds the scheduler's ServiceAccount to `system:auth-delegator`, which lets it check the callers of `/metrics`. Its `system:kube-scheduler` binding grants the same access, so `/metrics` works with this off too.
`scheduler.metrics.reader.create`	boolean	`true`	Creates the ClusterRole `{name}-metrics-reader`, which allows `get` on `/metrics`.
`scheduler.metrics.reader.serviceAccounts`	list	`[]`	The ServiceAccounts that may read `/metrics`, each with a `name` and a `namespace`. Needs `scheduler.metrics.reader.create`.
`scheduler.metrics.service.enabled`	boolean	`true`	Creates the Service `{name}-metrics` for the metrics port.
`scheduler.metrics.service.type`	string: `ClusterIP`, `NodePort` or `LoadBalancer`	`ClusterIP`	The Service's type.
`scheduler.metrics.service.port`	integer, 1 to 65535	`10259`	The Service's port, named `https-metrics`.
`scheduler.metrics.service.annotations`	map	`{}`	Annotations on the Service.
`scheduler.metrics.serviceMonitor.enabled`	boolean	`false`	Creates the ServiceMonitor `{name}`, which scrapes `/metrics` through the Service. Prometheus scrapes with its own ServiceAccount's token, so list that account in `scheduler.metrics.reader.serviceAccounts`. Needs the Prometheus Operator's ServiceMonitor CRD and `scheduler.metrics.service.enabled`.
`scheduler.metrics.serviceMonitor.interval`	string	`30s`	How often Prometheus scrapes the scheduler.
`scheduler.metrics.serviceMonitor.scrapeTimeout`	string	`10s`	The scrape's timeout.
`scheduler.metrics.serviceMonitor.additionalLabels`	map	`{}`	Extra labels on the ServiceMonitor, for example to match your Prometheus's `serviceMonitorSelector`.
`scheduler.metrics.serviceMonitor.tlsConfig`	map	`{}`	The scrape's TLS settings. When it's empty, the ServiceMonitor skips certificate verification, since the scheduler's certificate is self-signed.
`scheduler.metrics.serviceMonitor.relabelings`	list	`[]`	The endpoint's `relabelings`.
`scheduler.metrics.serviceMonitor.metricRelabelings`	list	`[]`	The endpoint's `metricRelabelings`.

Scheduler pods¶

These values shape the scheduler's own pods, which the default scheduler places.

Value	Type	Default	What it sets
`scheduler.verbosity`	integer	`2`	The log level, passed as `--v`. At `4`, the plugin logs its placement decisions, as Read the scheduler's logs shows.
`scheduler.probes.liveness`	map: `path`, `initialDelaySeconds`, `periodSeconds`	`/healthz`, `15`, `20`	The liveness probe, over HTTPS on the metrics port.
`scheduler.probes.readiness`	map: `path`, `initialDelaySeconds`	`/readyz`, `5`	The readiness probe, over HTTPS on the metrics port.
`scheduler.resources`	map	Requests of `200m` CPU and `4Gi` memory, limits of `2` CPUs and `12Gi` memory	The container's resources. The scheduler's memory use grows with the number of pods, nodes and PodGroups in the cluster.
`scheduler.nodeSelector`	map	`{}`	The pods' node selector.
`scheduler.affinity`	map	A required pod anti-affinity between the replicas on `kubernetes.io/hostname`	The pods' affinity. The default needs a separate node for each replica.
`scheduler.tolerations`	list	`[]`	The pods' tolerations.
`scheduler.topologySpreadConstraints`	list	A spread of the replicas across nodes, with `maxSkew: 1` and `whenUnsatisfiable: ScheduleAnyway`	The pods' topology spread constraints.
`scheduler.podAnnotations`	map	`{}`	Annotations on the pods. For annotation-based Prometheus discovery, set the scheme to `https`.

Invalid values¶

Helm refuses to install or upgrade the release, and prints the message, when the values break one of these rules:

Rule	Message
More than one replica needs leader election.	`scheduler.leaderElect must be true when scheduler.replicaCount is greater than 1`
While `scheduler.omeControllerIntegration.enabled` is on, the topology annotation is `ome.io/topology-key`.	`scheduler.plugin.podGroupTopologyKeyAnnotation must be ome.io/topology-key when scheduler.omeControllerIntegration.enabled is true; disable OME controller integration for generic producers`
With the PodDisruptionBudget on, `minAvailable` is below `scheduler.replicaCount`.	`scheduler.podDisruptionBudget.minAvailable must be less than scheduler.replicaCount so voluntary maintenance can make progress`
A ServiceMonitor needs the metrics Service.	`scheduler.metrics.serviceMonitor.enabled requires scheduler.metrics.service.enabled — the ServiceMonitor selects the metrics Service`

The scheduler checks the label keys in scheduler.plugin.topologyKey, scheduler.plugin.podGroupTopologyKeyAnnotation and scheduler.plugin.unsupportedPlacementGroupLabel when it starts. Helm installs a release with a malformed key. The scheduler then exits with an error that names the argument, like topologyKey "gpu clique" is not a valid node label key. Its pods restart until you fix the value.

PodGroups and labels¶

The plugin reads each gang from its pods and its PodGroup, a scheduler-plugins scheduling.x-k8s.io/v1alpha1 PodGroup. Install the PodGroup CRD from scheduler-plugins v0.35.4-devel, as Use the OME scheduler shows.

Key	On	What the plugin does with it
`scheduling.x-k8s.io/pod-group` label	Pod	Names the pod's PodGroup, in the pod's namespace. Pods with the same value form one gang, and a pod without the label is standalone.
`spec.minMember`	PodGroup	The gang's size. The plugin picks a domain once this many members exist, and binds them once this many reach Permit. It must be positive.
`spec.scheduleTimeoutSeconds`	PodGroup	How long the gang's members wait at Permit. When it's unset or not positive, `scheduler.plugin.defaultPermitTimeoutSeconds` applies.
`ome.io/topology-key` annotation	PodGroup	The node label whose values define the gang's accelerator domains. It wins over `scheduler.plugin.topologyKey`. `scheduler.plugin.podGroupTopologyKeyAnnotation` sets the annotation's name.
`ome.io/placement-group` label	Pod or PodGroup	Asks to place the gang with other gangs, which the plugin doesn't support, so it keeps the gang's members `Pending`. OME never sets it. `scheduler.plugin.unsupportedPlacementGroupLabel` sets the label's name.

The plugin writes nothing to PodGroups, including their status, and ignores their other fields, like spec.minResources.

When the PodGroup CRD is installed, OME creates a PodGroup for each multi-pod OMENative Instance, and copies the component's topologyKey into its ome.io/topology-key annotation. OME labels the Instance's pods even without the CRD, and the OME scheduler then keeps them Pending, as Scheduling events shows. How OME workloads use it has the details.

Scheduling events¶

When the scheduler places a pod, the pod gets a Scheduled event from ome-scheduler, or the name that you set in scheduler.name. When it can't, the pod gets FailedScheduling events whose message gives the reason. Next to kube-scheduler's own reasons, OMEGangPack gives these:

Reason	What it means
`no domain has room for gang {namespace}/{name}`	No domain has the capacity for the whole gang. The gang waits until one does.
`no domain has room for gang {namespace}/{name}: every fitting domain is reserved for a forming gang`	A domain has the capacity, but every domain that fits is reserved for a forming gang. The plugin wakes the gang's pods when a reservation drains or is released.
`PodGroup {namespace}/{name} not resolvable yet`	No PodGroup of that name exists in the pod's namespace. Create it, or fix the label. For OMENative pods, install the CRD and restart OME's manager, as Opt in OMENative components shows.
`PodGroup {namespace}/{name} must declare ome.io/topology-key or the scheduler must configure topologyKey`	The gang has no topology label. Set the component's `topologyKey`, or `scheduler.plugin.topologyKey`.
`waiting for all PodGroup member templates for {namespace}/{name}`	Fewer than `minMember` member pods exist.
`waiting for gang sibling {namespace}/{pod} to be placed`	The pod has a required pod affinity to a sibling, like its leader, and waits until the sibling is placed.
`pinned domain {domain} cannot fit all remaining gang members {namespace}/{name}`	Some members have nodes, and their domain has no room for the rest. The gang stays in that domain while any member runs there.
`adopted domain {domain} cannot fit remaining gang members {namespace}/{name}`	The same, after a scheduler restart or failover.
`pinned domain {domain} has no node feasible for gang member {namespace}/{name}`	No node in the gang's domain can run this member and leave room for the rest.
`gang {namespace}/{name} already has members in multiple topology domains`	The gang's placed members span domains or unlabeled nodes, for example after node labels changed. Its other pods wait until the split ends.
`node is reserved for a forming gang`	The node is in the domain of a forming gang. Standalone pods can use it once that gang's last member has a node, or the gang is released.
`gang reservation released after all candidate nodes were filtered`	Another filter rejected all the nodes the gang could use in its domain. The scheduler tries its other domains first: the gang's first veto in a domain, while another untried one has room, wakes the vetoed pod to retry, as the `domain_failed` trigger in Metrics counts. Once every domain with room has failed, the gang's next attempt tries them all afresh.
`gang {namespace}/{name} unwound after a member failed to schedule`	Another member of the pod's gang failed, so the scheduler tries the gang again.
`PodGroup {namespace}/{name} must declare a positive minMember`	Set a positive `minMember` on the PodGroup.
`partner placement groups are not supported`	Remove the `ome.io/placement-group` label from the pod and its PodGroup.

Nodes that the message counts as didn't satisfy plugin(s) [OMEGangPack] are outside the gang's domain, or can't run the member and leave room for the rest.

Metrics¶

Each replica serves kube-scheduler's standard metrics and OMEGangPack's metrics at /metrics. The Metrics endpoint values set who may read them, and Use the OME scheduler sets up a Prometheus scrape. Only the leader schedules, so only its OMEGangPack metrics change. OMEGangPack's metrics are alpha: their names and labels can change.

Metric	Type	Labels	What it counts
`ome_scheduler_gang_pin_total`	Counter	`result`	Domain decisions for gangs, by `result`, as the next table lists. A gang still waiting for a domain adds a `no_fit` or a `reserved` each time the scheduler tries one of its pods.
`ome_scheduler_gang_gate_total`	Counter	`result`	Permit checks of gang members, by `result`: `wait` while the rest of the gang still needs nodes, and `admit` when a member completes the gang and lets it bind.
`ome_scheduler_gang_activation_total`	Counter	`trigger`	Times the plugin sent pods back to the scheduling queue, by `trigger`, as the trigger table lists. A wake-up counts once, however many pods it moves.
`ome_scheduler_gang_unwind_total`	Counter	None	Times the plugin released a gang's pin and rejected its members waiting at Permit, for example after a member failed to schedule. It also grows when a gang ends, its PodGroup is deleted or re-created, or its topology label changes, so it isn't a failure count on its own.
`ome_scheduler_pinned_groups`	Gauge	None	Gangs pinned to a domain now, including gangs whose members all have nodes.

The result values of ome_scheduler_gang_pin_total:

`result`	When the plugin counts it
`pinned`	It picked a domain with room for the whole gang, and pinned the gang to it.
`no_fit`	No domain has the capacity for the gang.
`reserved`	A domain has the capacity, but every domain that fits is reserved for a forming gang. The pod parks until a reservation drains or is released.
`adopted`	It pinned a gang to the domain where its members already have nodes, for example after a restart or a failover.
`stale_replan`	It released a gang's pin to pick again, because the domain lost its nodes, or lost room for the gang while no member had a node there.
`topology_replan`	It released a gang's pin to pick again, because the PodGroup's topology label changed.

The trigger values of ome_scheduler_gang_activation_total:

`trigger`	When the plugin counts it
`permit`	A gang member reached Permit, and the gang's other pods retry. A member with a required pod affinity to it, like a worker to its leader, can now place.
`templates_complete`	A waiting gang reached `minMember` pods, and the members that parked while it was short retry.
`reservation_released`	A forming gang's domain reservation drained, when its last member got a node, or was released. The plugin wakes every pod that a reservation parked: standalone pods that hit `node is reserved for a forming gang`, and gangs whose fitting domains were all reserved.
`podgroup_change`	The plugin's PodGroup informer stored a new or edited PodGroup, and the gang's members retry against it. This wakes a member that retried before the plugin saw the change, which would otherwise stay parked.
`domain_failed`	Another filter rejected a pinned gang member on every node of its gang's domain, the gang's first veto in that domain, while another untried domain had room when the gang was pinned. The member retries after its backoff, and the retry plans around the domains that failed; without the wake-up, it would wait for the scheduler's periodic retry of unschedulable pods, since the rejecting filters raise no event for it. This wake-up moves only the vetoed member. A veto in the only untried domain that fit the gang, or in the domain its bound members keep it in, wakes nothing.

Scheduler profile¶

The chart writes the scheduler's KubeSchedulerConfiguration to the config.yaml key of the ConfigMap {name}-config. With the chart's defaults, for a release in the ome namespace, it renders as follows, without its comments:

config.yaml

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
  resourceName: ome-scheduler
  resourceNamespace: ome
profiles:
  - schedulerName: ome-scheduler
    plugins:
      multiPoint:
        enabled:
          - name: OMEGangPack
        disabled:
          - name: DefaultPreemption
      score:
        enabled:
          - name: NodeResourcesFit
            weight: 10
          - name: OMEGangPack
            weight: 10
        disabled:
          - name: PodTopologySpread
    pluginConfig:
      - name: OMEGangPack
        args:
          podGroupTopologyKeyAnnotation: "ome.io/topology-key"
          unsupportedPlacementGroupLabel: "ome.io/placement-group"
          defaultPermitTimeoutSeconds: 600
          podGroupSyncTimeoutSeconds: 30
          gcIntervalSeconds: 60
          standaloneDomainPacking: true
      - name: NodeResourcesFit
        args:
          scoringStrategy:
            type: MostAllocated
            resources:
              - name: cpu
                weight: 1
              - name: memory
                weight: 1

NodeResourcesFit always uses the MostAllocated strategy, which favors busy nodes, where kube-scheduler's default, LeastAllocated, favors empty ones. kube-scheduler's other default plugins stay on, with their default weights. When a Helm upgrade changes this configuration, the Deployment replaces the scheduler's pods.

Alfred simulates the OME scheduler from a copy of this configuration, as Write the scheduler configurations shows.

The OME scheduler: why OME has a second scheduler, and how OMEGangPack places a gang.
Use the OME scheduler: install the chart, opt a workload in, tune packing and troubleshoot.
Gang scheduling: the PodGroups that OME creates for multi-pod Instances.
Spread Instances across fault domains: the hard spread constraints that the OME scheduler enforces.
Let Alfred migrate Instances: to migrate a multi-pod Instance, Alfred needs its pods on the OME scheduler.