Set accelerator quotas Unreleased: coming in v1.3

An AcceleratorQuota tree divides a cluster's accelerators among teams. The quota manager, ome-quota-manager, checks the tree and turns it into Kueue queues, which admit each team's workloads against its budget.

You install the quota manager and give 24 A100 GPUs to the group ml-research. You split them: 16 for team-alpha, and 8 for team-beta, which can borrow up to 4 more while team-alpha leaves them idle. Then you charge an InferenceService to a team.

How quotas work¶

AcceleratorQuotas, short name aq, are cluster-scoped and form a tree under root, which the quota manager creates. Every other node names its parent in spec.parentRef.name, and has one of two roles in spec.role:

• A Cohort groups other nodes, whose budgets add up to at most its own. It isn't a cap at run time: a team that borrows can take the group past it.
• A ClusterQueue is a team, with no children. Its workloads charge its budgets.

A budget in spec.budgets counts one resource on one Kueue ResourceFlavor, such as nvidia.com/gpu on a100. A Cohort's budget sets only nominal. A team's budget has three amounts:

A team's priorityTier has no effect in this release.

The quota manager writes a Cohort node to Kueue as a Cohort, and a team as a ClusterQueue with a LocalQueue in each enrolled namespace. These objects take their node's name and carry the label ome.io/quota-managed-by: ome-quota-manager.

The webhook denies a change that breaks the tree's rules. A node can also break one later, when a budget is more than the peak capacity that the quota manager measures. The quota manager then freezes the node and its descendants: their Kueue objects stay as they were, so admitted workloads keep running. Each budget is checked against capacity on its own, so to cap the total, give root a budget.

The tree's depth is bounded: a node can be at most quotaManager.maxTreeDepth edges below root, 5 by default, counting root as 0, so ml-research sits at 1 and the teams at 2. The bound is a guardrail against accidental sprawl, not a Kueue limit. The webhook denies a node past it, naming both numbers: node is 6 levels below the root, exceeding the configured maximum of 5. To nest deeper, raise quotaManager.maxTreeDepth in quota-values.yaml and upgrade the release, as in Lower the high-water mark: the chart passes the value to the quota manager as its --max-tree-depth flag, and 0 turns the check off. Lowering the bound under a stored node's depth doesn't remove the node: it reports DepthExceeded and is frozen, like any node that breaks a rule.

The quota manager itself runs in one of two modes, and an install runs exactly one: the two write different halves of root's status. This page covers workload, which turns this cluster's tree into the Kueue objects above. The other mode, management, holds a fleet-wide tree and projects each cluster's share onto workload clusters. Its reconciliation is still in development, so don't build on it, or on the fields that exist for it: spec.distribution and budgets[].perCluster in the API, and the projection and remoteAccess values in the chart.

Step 1: Install the quota manager¶

The quota manager has no published image, so build one in your OME checkout. The binary is static, so an image built FROM scratch can hold it:

mkdir -p quota-image
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-s -w" \
  -o quota-image/ome-quota-manager ./cmd/ome-quota-manager
cat > quota-image/Dockerfile <<'EOF'
FROM scratch
COPY ome-quota-manager /ome-quota-manager
USER 65532:65532
ENTRYPOINT ["/ome-quota-manager"]
EOF
docker build --platform linux/amd64 -t registry.example.com/ome/ome-quota-manager:dev quota-image
docker push registry.example.com/ome/ome-quota-manager:dev

docker push uploads the image, tagged dev like the images from Install from source. For Arm nodes, set GOARCH=arm64 and --platform linux/arm64. The chart sets runAsNonRoot without a user ID, so the image needs its numeric USER.

Write the values. mode picks one of the two modes from How quotas work, here workload. It has no default: the chart fails to render without it, with an error that ends in quotaManager.mode must be "workload" or "management", got "". enrolledNamespaces lists the namespaces that the quotas govern. While it's empty, as it is by default, the quota manager writes nothing to Kueue:

global:
  hub: registry.example.com/ome
quotaManager:
  mode: workload
  image:
    tag: dev
  materialize:
    enrolledNamespaces:
      - ml-serving

The chart then pulls registry.example.com/ome/ome-quota-manager:dev. For a registry that needs credentials, see Install from a private registry.

Install the chart:

helm install ome-quota-manager ./charts/ome-quota-manager \
  --namespace ome -f quota-values.yaml --server-side=false

helm install ome-quota-manager ./charts/ome-quota-manager \
  --namespace ome -f quota-values.yaml

The release notes warn that the chart isn't installing the AcceleratorQuota CRD, and offer a value, quotaManager.crd.install, for a cluster where nothing else does. Leave it at its default, false: ome-crd installed the CRD here, as on any cluster that runs OME, and the CRD is one cluster-scoped object, so whichever release claims it second — this chart with crd.install: true, or an ome-crd installed later — fails Helm's ownership check.

Wait for both replicas:

kubectl -n ome rollout status deployment/ome-quota-manager

deployment "ome-quota-manager" successfully rolled out

The quota manager then creates root:

kubectl get aq -o custom-columns=NAME:.metadata.name,ROLE:.spec.role,PARENT:.spec.parentRef.name

NAME   ROLE     PARENT
root   Cohort   <none>

Step 2: Create the quota tree¶

The quota manager only reads Kueue ResourceFlavors, so create one that matches the A100 nodes:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
  name: a100
spec:
  nodeLabels:
    nvidia.com/gpu.product: NVIDIA-A100-SXM4-80GB

kubectl apply -f a100-flavor.yaml

resourceflavor.kueue.x-k8s.io/a100 created

The quota manager measures capacity when nodes change, and at least once a minute by default: see Check that capacity reporting is alive. It reports the measurement on root:

kubectl get aq root -o jsonpath='{range .status.capacity[*]}{.resourceName}{" "}{.resourceFlavor}{" "}{.allocatable}{" "}{.highWaterMark}{"\n"}{end}'

google.com/tpu a100 0 0
nvidia.com/gpu a100 32 32

The columns are a resource, a flavor, the accelerators on nodes that are Ready and not cordoned, and the high-water mark that budgets are checked against. The mark also counts cordoned and not-Ready nodes, so it stays put while you drain nodes: Lower the high-water mark explains when it comes down. Measured resources appear on all flavors, so google.com/tpu shows 0.

Write the tree once root lists your resource and flavor. Until then, a budget for them can report CapacityExceeded. The tree puts the Cohort ml-research under root, with two teams under it:

apiVersion: ome.io/v1beta1
kind: AcceleratorQuota
metadata:
  name: ml-research
spec:
  role: Cohort
  parentRef:
    name: root
  budgets:
    - resourceName: nvidia.com/gpu
      resourceFlavor: a100
      nominal: "24"
---
apiVersion: ome.io/v1beta1
kind: AcceleratorQuota
metadata:
  name: team-alpha
spec:
  role: ClusterQueue
  parentRef:
    name: ml-research
  budgets:
    - resourceName: nvidia.com/gpu
      resourceFlavor: a100
      nominal: "16"
---
apiVersion: ome.io/v1beta1
kind: AcceleratorQuota
metadata:
  name: team-beta
spec:
  role: ClusterQueue
  parentRef:
    name: ml-research
  budgets:
    - resourceName: nvidia.com/gpu
      resourceFlavor: a100
      nominal: "8"
      borrowingLimit: "4"

ml-research comes first: the webhook denies a team whose parent doesn't exist yet.

Apply the tree:

kubectl apply -f quota-tree.yaml

acceleratorquota.ome.io/ml-research created
acceleratorquota.ome.io/team-alpha created
acceleratorquota.ome.io/team-beta created

Check that the nodes are valid and written to Kueue:

kubectl get aq -o custom-columns='NAME:.metadata.name,ROLE:.spec.role,PARENT:.spec.parentRef.name,READY:.status.conditions[?(@.type=="Ready")].status,MATERIALIZED:.status.conditions[?(@.type=="Materialized")].status'

NAME          ROLE           PARENT        READY   MATERIALIZED
ml-research   Cohort         root          True    True
root          Cohort         <none>        True    True
team-alpha    ClusterQueue   ml-research   True    True
team-beta     ClusterQueue   ml-research   True    True

If a node shows False, see A node isn't Ready or Materialized.

The quota manager has written the teams' ClusterQueues:

kubectl get clusterqueues.kueue.x-k8s.io -l ome.io/quota-managed-by=ome-quota-manager -o custom-columns=NAME:.metadata.name,COHORT:.spec.cohortName

NAME         COHORT
team-alpha   ml-research
team-beta    ml-research

It has also written their LocalQueues in ml-serving:

kubectl get localqueues.kueue.x-k8s.io -n ml-serving -l ome.io/quota-managed-by=ome-quota-manager -o custom-columns=NAME:.metadata.name,CLUSTERQUEUE:.spec.clusterQueue

NAME         CLUSTERQUEUE
team-alpha   team-alpha
team-beta    team-beta

Step 3: Charge workloads to a team¶

To charge an InferenceService in ml-serving to team-beta, label it with the team's name:

metadata:
  labels:
    kueue.x-k8s.io/queue-name: team-beta

OME copies the label to the pods of all components, and Kueue keeps the pods SchedulingGated until team-beta's ClusterQueue admits them.

On an OMENative component, OME keeps the label only when the pods request an accelerator in ome.controller.quotaAcceleratorResources, nvidia.com/gpu or google.com/tpu by default. The wait for admission doesn't count against the readiness deadline.

Kueue admits each pod on its own, so a nearly full budget can admit part of a multi-node Instance. Size budgets in whole Instances, with room for an update, which by default starts new pods before the old ones drain.

To move a workload to another team, change the label. On OMENative, running pods keep charging the old team until the next rollout replaces them.

Once Kueue admits the pods, kubectl get aq team-beta shows their GPUs under ADMITTED, next to the team's share under NOMINAL.

Change the tree¶

Edit quota-tree.yaml and apply it again. To check a change without writing it, add --dry-run=server: the webhook still runs, and kubectl marks the objects (server dry run).

Raising team-alpha to 20, for example, would bring the children of ml-research to 28, more than its 24. The apply fails with an error that ends in admission webhook "acceleratorquota.ome-quota-manager.validator" denied the request: ml-research: budget nvidia.com/gpu on a100 is 24 but its children total 28.

To change a node's role, delete the node and create it again: changing spec.role in place leaves the old Kueue object behind.

Lower the high-water mark¶

Each budget is checked against its resource and flavor's high-water mark on root, not against the capacity of the moment. The mark is a running peak of the installed capacity: the accelerators on every node, the cordoned and not-Ready ones included. It's damped, so it ignores dips that say nothing about what the teams are entitled to, such as a drain, a rolling reboot or a restarting device plugin:

• Growth is believed at once: a measurement above the mark raises it to the new installed capacity.
• A drop is believed only when the installed capacity falls below the mark by more than the band: quotaManager.capacity.hysteresisPercent percent of the mark, 10 by default, which the chart passes to the quota manager as its --capacity-hysteresis-percent flag. The mark then falls all the way to the new installed capacity, not to the edge of the band.

So the mark comes down within a minute when a decommission takes the installed capacity past the band. Deleting one of the four A100 nodes leaves 24 GPUs installed, 25% below the mark of 32, so the mark falls to 24, and a budget over 24 reports CapacityExceeded. But the band is a magnitude, not a wait: retire 2 of the 32 GPUs instead, 6.25%, and the mark stays at 32 for good, so the tree keeps accepting budgets for 2 GPUs that are gone. The marks live in root's status, so restarting the quota manager doesn't lower them either.

To bring the mark down after a decommission within the band, turn the damping off and let the quota manager measure. Set hysteresisPercent to 0, keeping the other values in the file:

global:
  hub: registry.example.com/ome
quotaManager:
  mode: workload
  image:
    tag: dev
  capacity:
    hysteresisPercent: 0
  materialize:
    enrolledNamespaces:
      - ml-serving

helm upgrade ome-quota-manager ./charts/ome-quota-manager \
  --namespace ome -f quota-values.yaml

Helm reports the upgrade, and the quota manager's pods restart with the damping off. Within a minute, the mark matches the installed capacity:

kubectl get aq root -o jsonpath='{range .status.capacity[*]}{.resourceName}{" "}{.resourceFlavor}{" "}{.allocatable}{" "}{.highWaterMark}{"\n"}{end}'

google.com/tpu a100 0 0
nvidia.com/gpu a100 30 30

Then set hysteresisPercent back to 10, or remove it from the file, and upgrade again.

A pair that stops being measured at all, because the last node that carried it is gone, you delete its ResourceFlavor, or you remove its resource from quotaManager.capacity.resources, keeps its last mark and reports 0 allocatable, whatever the band. Turning the damping off doesn't lower this mark either: the band judges measurements, and a pool with no hardware isn't measured at zero, it isn't measured at all, so no reading arrives for the mark to follow down. The entry stays in root's status as the record of what the pair once held, and the mark moves again only when nodes report the pair again: at once when the new installed capacity is above it, past the band when it's below. So when you retire a pool for good, also delete the budgets that name its resource and flavor. Each one keeps passing its check against the stale mark, promising teams accelerators that no node has.

Check that capacity reporting is alive¶

On stable hardware, the numbers on root stop changing, so they can't tell a healthy cluster from a quota manager that stopped measuring. What tells the two apart is observedAt on each entry of root's status.capacity, the time of the last measurement:

kubectl get aq root -o jsonpath='{range .status.capacity[*]}{.resourceName}{" "}{.resourceFlavor}{" "}{.observedAt}{"\n"}{end}'

google.com/tpu a100 2026-10-05T09:41:12Z
nvidia.com/gpu a100 2026-10-05T09:41:12Z

The quota manager measures on a timer: the smaller of quotaManager.resyncInterval, 10 minutes by default, and the report interval, quotaManager.capacity.reportInterval, 1 minute by default, which the chart passes to the quota manager as its --capacity-report-interval flag. It also measures at once when an AcceleratorQuota is written, or when a node is added or deleted, cordoned or uncordoned, relabeled, or changes its readiness or its accelerators. Creating a ResourceFlavor triggers no measurement, so a new flavor shows up on root with the next one: the wait before the tree in Step 2.

A measurement that changes what root reports is written at once. The report interval bounds the rest: a measurement that finds the stamps a report interval old writes them anew, even though the numbers stand. So with the defaults, a quota manager that measures keeps every observedAt within about a minute of now, and the stamps say that the reporting is alive, not that the hardware changed.

A stamp well past the report interval means that the measuring stopped, and with it everything this page describes: nothing raises the mark for new nodes, or lowers it for retired ones. A measurement that fails, such as a node list the API server refuses, doesn't refresh the stamps either, so check the quota manager's pods and their logs. Setting reportInterval to "" or 0s turns the refresh off, and the timer falls back to resyncInterval: observedAt then dates the last change of the numbers, and a reporter that stopped no longer shows.

Budget other accelerators¶

Two lists, one in each chart, decide which resources count as accelerators:

ome.controller.acceleratorResources, without quota, is a different list, unrelated to quotas.

To budget AMD GPUs as well, add amd.com/gpu to both lists. In the quota manager's values:

global:
  hub: registry.example.com/ome
quotaManager:
  mode: workload
  image:
    tag: dev
  capacity:
    resources:
      - amd.com/gpu
      - google.com/tpu
      - nvidia.com/gpu
  materialize:
    enrolledNamespaces:
      - ml-serving

helm upgrade ome-quota-manager ./charts/ome-quota-manager \
  --namespace ome -f quota-values.yaml

Helm reports the upgrade, and the quota manager's pods restart with the new list.

Then add it to the ome-resources values. Keep the values from Install from source in the file, because helm upgrade -f resets every value that the file leaves out:

global:
  hub: registry.example.com/ome
ome:
  controller:
    tag: dev
    quotaAcceleratorResources:
      - nvidia.com/gpu
      - google.com/tpu
      - amd.com/gpu
  omeAgent:
    tag: dev
modelAgent:
  enabled: true
  image:
    tag: dev

helm upgrade ome ./charts/ome-resources --namespace ome -f values.yaml

Helm reports the upgrade, and the controller pods restart with the new list.

Metrics¶

The quota manager serves Prometheus metrics on port 8080, at /metrics. OME's bundled Prometheus scrapes it when both releases are in ome: see Collect metrics. For the Prometheus Operator, set quotaManager.metrics.serviceMonitor.enabled: true, and put the labels that your Prometheus selects in quotaManager.metrics.serviceMonitor.additionalLabels.

On the budget gauges, plane is the quota manager's mode, so on this page's install it's always workload. The label exists because a management-mode install publishes the same four series with other meanings, nominal being the authored fleet total rather than one cluster's share, so a query must never sum the two planes.

The capacity gauges are a snapshot of the last measurement, rewritten whole, so a series disappears when its cause does: create the flavor that NoMatchingFlavor points at, and the unattributed series are gone with the next measurement. The two counters say whether the writing to Kueue is alive. ome_quota_backend_applied_total keeps rising while the quota manager materializes, even on a cluster where nothing changes; a counter that stands still means that nothing is written, because enrolledNamespaces is empty, every node is frozen or the quota manager stopped. ome_quota_backend_swept_total moves only when a sweep deletes something, so standing still is its steady state.

Let a group manage quotas¶

A ClusterRoleBinding to the built-in view, edit or admin role already lets users read AcceleratorQuotas. To let a group edit the tree, bind it to a ClusterRole of its own. Leave out acceleratorquotas/status, which only the quota manager writes:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: accelerator-quota-admin
rules:
  - apiGroups: ["ome.io"]
    resources: ["acceleratorquotas"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: accelerator-quota-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: accelerator-quota-admin
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: Group
    name: quota-admins

kubectl apply -f accelerator-quota-admin.yaml

clusterrole.rbac.authorization.k8s.io/accelerator-quota-admin created
clusterrolebinding.rbac.authorization.k8s.io/accelerator-quota-admin created

Troubleshooting¶

kubectl apply fails with a webhook error¶

A denial names the node and the rule that the change breaks. A failure to call the webhook means that no quota manager pod is ready. The webhook fails closed, so every create, update and delete of an AcceleratorQuota fails until one is.

kubectl -n ome get pods shows why the pods aren't ready. A cause outside the webhook, such as an image the nodes can't pull, you fix as usual, and admission returns with the first ready pod. But the webhook itself can be the cause: a pod reports ready only once its webhook serves, so when the webhook's certs are broken — the Secret ome-quota-manager-webhook-cert deleted, for example, which the quota manager updates but never creates — no pod can become ready, and the wedge doesn't clear on its own.

To recover, disable the webhook: set quotaManager.webhook.enabled to false, keeping the other values in the file, and upgrade the release. The webhook gates only AcceleratorQuotas, so the wedge doesn't block the upgrade itself:

global:
  hub: registry.example.com/ome
quotaManager:
  mode: workload
  image:
    tag: dev
  materialize:
    enrolledNamespaces:
      - ml-serving
  webhook:
    enabled: false

helm upgrade ome-quota-manager ./charts/ome-quota-manager \
  --namespace ome -f quota-values.yaml

Helm reports the upgrade. It removes the ValidatingWebhookConfiguration ome-quota-manager.ome.io, so quota writes go through at once, and the pods restart without the webhook or its certs, so they become ready and the quota manager reconciles again.

Fix the cause, then set enabled back to true, or remove webhook from the file, and upgrade again. The upgrade restores the webhook's objects, and for broken certs it is the fix: the chart recreates the Secret empty when it's gone, and on startup the quota manager generates the serving certificate, injects the CA bundle into the new ValidatingWebhookConfiguration, and reports ready once the webhook serves. The restored webhook fails closed from the moment the upgrade creates it, so quota writes fail again until a pod is ready. Wait for the rollout:

kubectl -n ome rollout status deployment/ome-quota-manager

deployment "ome-quota-manager" successfully rolled out

Then confirm that admission answers:

kubectl apply -f quota-tree.yaml --dry-run=server

acceleratorquota.ome.io/ml-research unchanged (server dry run)
acceleratorquota.ome.io/team-alpha unchanged (server dry run)
acceleratorquota.ome.io/team-beta unchanged (server dry run)

Pods stay SchedulingGated¶

kubectl describe workloads -n ml-serving shows why Kueue holds a pod. The usual causes:

• The team's quota is in use, by its own pods or by a team that borrowed it.
• The Workload reports LocalQueue team-beta doesn't exist. Add the namespace to enrolledNamespaces.
• The pod requests a resource missing from its ClusterQueue, such as an RDMA device.

A ClusterQueue lists its team's accelerators, plus CPU, memory and ephemeral storage with ceilings too high to limit anyone. To admit another resource, give it a ceiling in coverResources, and upgrade the release as in Budget other accelerators. Helm merges the map, so the defaults stay:

quotaManager:
  materialize:
    coverResources:
      rdma/hca_shared_devices_a: 1M

Pods run without charging a budget¶

Check that the pods carry kueue.x-k8s.io/queue-name, and that Kueue's pod integration is on. On OMENative, OME removes the label from pods that request no listed accelerator: see Budget other accelerators.

A node isn't Ready or Materialized¶

Read the node's conditions with their messages:

kubectl get aq team-beta -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{": "}{.message}{"\n"}{end}'

Ready=True Admitted: node is a valid member of the quota tree
Degraded=False Admitted: no invariant violated
Materialized=True Admitted: enforcement objects match this node's budget

Degraded is the opposite of Ready, with the same reason. Materialized is set only while enrolledNamespaces lists a namespace. The reasons for False:

Clean up¶

Delete the workloads that charge the teams, then the teams:

kubectl delete aq team-alpha team-beta

acceleratorquota.ome.io "team-alpha" deleted
acceleratorquota.ome.io "team-beta" deleted

Then delete the Cohort. The webhook denies deleting a node that still has children:

kubectl delete aq ml-research

acceleratorquota.ome.io "ml-research" deleted

Uninstall the chart:

helm uninstall ome-quota-manager -n ome

Helm reports that it kept the Secret ome-quota-manager-webhook-cert, and that the release is uninstalled.

Four objects remain: root, the Kueue Cohort root, the webhook Secret and the ResourceFlavor. With the quota manager gone, clear the finalizer on root yourself, then delete all four:

kubectl patch aq root --type=merge -p '{"metadata":{"finalizers":null}}'
kubectl delete aq root
kubectl delete cohorts.kueue.x-k8s.io root
kubectl delete secret -n ome ome-quota-manager-webhook-cert
kubectl delete resourceflavors.kueue.x-k8s.io a100

acceleratorquota.ome.io/root patched
acceleratorquota.ome.io "root" deleted
cohort.kueue.x-k8s.io "root" deleted
secret "ome-quota-manager-webhook-cert" deleted
resourceflavor.kueue.x-k8s.io "a100" deleted

Next steps¶

• kubectl ome quota: kubectl ome quota tree prints the tree, and validate exits with status 2 when it breaks a rule.
• Use the OME scheduler: OME's alpha second scheduler, for Kubernetes 1.35, places each opted-in gang in one topology domain.