Tune routing capacity polling Preview Unreleased: coming in v1.3

A capacity poll lets a workload cluster cap its share of an InferenceService's traffic at what it can serve. The control plane asks the cluster how many of the service's replicas can serve now, and counts no more than that in the cluster's TrafficMap weight. Use it when a cluster knows better than the control plane what can serve: for example, how many of its engine and decoder pairs work together during a rollout. A report can lower a cluster's weight but never raise it, and a failed poll leaves the weight as it would be without one.

Multi-cluster routing is alpha, still in development, and off by default. Its API, the capacity settings included, can change without notice. You turn it on with the ome-resources chart's values ome.multicluster.enabled, ome.multicluster.role and ome.multicluster.config.routing.enabled, as Step 1 shows.

How the poll caps a cluster's weight¶

A workload cluster that has admitted the InferenceService and reports an endpoint for it is a home, with an entry in the TrafficMap. OME polls a home every period, keeps its last samples readings, and takes the quorum-th lowest as its ceiling. With quorum: 2, the readings 4, 1 and 4 give a ceiling of 4, since one low reading can't set it alone, and 1, 4 and 1 give 1.

The entry's capacity.allocated becomes the lower of the admitted replicas and the ceiling, and the weight follows from it. A ceiling of 0 takes all of the home's traffic away, even with a probe's allFailedPolicy: PreserveTraffic.

After a failed poll, the home has no ceiling, and goes back to its admitted replicas until quorum polls in a row succeed. So a network fault can leave a home with more traffic than it can serve for a while, but can't take its traffic away. A home also starts over with no readings when the controller restarts or changes leader, when you change a capacity setting, or when its endpoint changes.

The report format¶

A poll is one request, with the poll's method and no body, to the entry's endpoint followed by the poll's path. The endpoint is the status.url of the service's copy on that workload cluster, so the poll URL depends on the cluster's ingress. For worker-a:

OME's HTTPRoute or Ingress sends all paths under the endpoint to the model server, which answers the poll with a 404. So add a route of your own that sends the poll URL to your reporter. For an HTTPRoute on the same Gateway, the Gateway API gives your longer path precedence over OME's. For a separate Ingress, precedence depends on your Ingress controller.

Your reporter answers with status 200 and a JSON body in the Report format, the only one that OME reads:

{"servable": 4, "observedAt": "2026-09-28T10:00:00Z"}

• Count replicas, not utilization or request slots: OME compares the count with the admitted replicas.
• Answer at the path itself, with no login: the poll sends no credentials, and OME doesn't follow redirects.
• Keep the reporter's clock in step with the control plane's.

Step 1: Set the operator defaults¶

Set the poll for all routed InferenceServices under ome.multicluster.config.routing.capacity, in your ome-resources values file. Do this once they all have reporters: a service without one falls back on all its homes, and keeps its CapacityFallback condition True. To poll only some services, leave the block out, and give each a poll as in Step 2.

This example turns routing on, with the poll from the chart's comments:

ome:
  multicluster:
    enabled: true
    role: control-plane
    config:
      routing:
        enabled: true
        capacity:
          path: /capacity
          method: GET
          format: Report
          samples: 10
          quorum: 2
          period: 10s
          timeout: 3s
          maxAge: 30s

Keep your other values, such as a routing health probe, in the same file: helm upgrade resets the values that aren't in it to the chart's defaults.

Once you set path, every field but options is required, or the controller doesn't start: see The controller rollout doesn't finish.

To choose the values:

• Raise quorum when successive polls can reach different reporters behind a load balancer. A drop then applies only when quorum readings agree, after at least quorum polls.
• Give maxAge room above quorum times period for the time that the reporter holds a figure before it answers. A reporter that freezes loses its home's ceiling within maxAge.

Upgrade the release with the chart version that you already run, 1.3.0 or later:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that the release ome has been upgraded, and the controller restarts with the new settings. Check that it's back up:

kubectl rollout status deployment ome-controller-manager -n ome

When the rollout finishes, kubectl prints:

deployment "ome-controller-manager" successfully rolled out

Step 2: Set a poll for one service¶

Give one InferenceService its own poll in its spec.routing.capacity, with or without a poll in the chart. It replaces the chart's poll as a whole, so set every field but options. For endpoints that spread polls across several reporters, this poll keeps four readings, needs three to agree before it lowers a ceiling, and lets a reading count for four periods:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-0-6b
spec:
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 2
    maxReplicas: 2
  placement:
    policy: ClusterAffinity
    mode: All
    clusterAffinity:
      - matchFields:
          - key: metadata.name
            operator: In
            values: [worker-a, worker-b]
  routing:
    capacity:
      path: /capacity
      method: GET
      format: Report
      samples: 4
      quorum: 3
      period: 10s
      timeout: 3s
      maxAge: 40s

The API server rejects a poll that's missing a field, with the message an enabled capacity poll must specify path, method, format, period, timeout, samples, quorum, and maxAge. It accepts some other mistakes: see The TrafficMap doesn't pick up the poll.

kubectl apply -f isvc.yaml

inferenceservice.ome.io/qwen3-0-6b configured

The new settings start the homes over with no readings, so they get their ceilings from the third poll on, about 20 seconds later.

Step 3: Watch the capacity state¶

Check the TrafficMap, which has the InferenceService's name and namespace. Its CapacityFallback condition sums up the homes:

kubectl get trafficmap qwen3-0-6b -n qwen3-0-6b \
  -o 'custom-columns=FALLBACK:.status.conditions[?(@.type=="CapacityFallback")].status,REASON:.status.conditions[?(@.type=="CapacityFallback")].reason,MESSAGE:.status.conditions[?(@.type=="CapacityFallback")].message'

FALLBACK   REASON                      MESSAGE
False      EndpointCapacityAvailable   endpoint capacity is available for all 2 home(s)

While any home has no ceiling, it's True with the reason EndpointCapacityUnavailable: see A home falls back to its admitted replicas. Traffic map lists its other reasons.

List each home's weight and capacity:

kubectl get trafficmap qwen3-0-6b -n qwen3-0-6b \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{" "}{.weight}{" "}{.capacity.allocated}{" "}{.capacity.ready}{" "}{.capacity.source}{" "}{.capacity.reported}{"\n"}{end}'

When worker-a reports 3 and worker-b reports 1, and both have two ready replicas:

worker-a 2 2 2 ControlPlane 3
worker-b 1 1 2 Endpoint 1

On worker-b, the ceiling of 1 lowers allocated from 2 to 1, so source is Endpoint. On worker-a, the ceiling of 3 is above the two admitted replicas, so source stays ControlPlane. Its reported value is 3, which a failed poll would leave empty. The weights 2 and 1 give worker-a two thirds of the traffic.

OME leaves out allocated and ready when they're 0, so a missing number means 0. Traffic map describes the fields.

While the poll is on, the gauge ome_trafficmap_capacity_fallback_homes, with the labels namespace and trafficmap, counts the homes without a ceiling. Alert when it stays above 0.

kubectl ome placement endpoint shows the condition and a code for each home's fallback reason and, with -o wide, its capacity source. The command reads alpha features.

Turn the poll off¶

To turn the poll off for one InferenceService, whatever the chart sets, replace the capacity block in isvc.yaml with disabled: true:

spec:
  routing:
    capacity:
      disabled: true

Apply the file again:

kubectl apply -f isvc.yaml

inferenceservice.ome.io/qwen3-0-6b configured

OME stops polling the service's homes, and CapacityFallback turns False with the reason CapacityPollingDisabled. Don't use spec.routing.enabled: false for this: it takes the service out of routing, and OME deletes its TrafficMap.

To stop the chart's poll, remove the capacity block from your values file, and run helm upgrade again. OME keeps polling for InferenceServices that set their own spec.routing.capacity.

The observer limits¶

One chart block, ome.multicluster.config.routing.observer, bounds the work that the capacity polls and the routing health probes create, across all routed InferenceServices. These are the chart's defaults:

ome:
  multicluster:
    config:
      routing:
        observer:
          maxConcurrentReconciles: 8
          maxConcurrentRequests: 16
          maxResponseBytes: 65536
          minPeriod: 1s
          maxSamples: 100

With routing.enabled: true, all five limits are required, even when the chart sets no poll or probe. A limit that you leave out of your values file keeps its default, but one set to 0, or removed with null, stops the controller from starting, like a bad poll: see The controller rollout doesn't finish. OME checks the chart's poll and probe against minPeriod and maxSamples when the controller starts, and a spec.routing block when it picks the block up, where a broken bound freezes the TrafficMap like any other mistake in the block.

A poll or probe waits for one of the maxConcurrentRequests slots before it's sent. The wait can't fail it, since its timeout starts with the request. But when many homes share the budget, the time between a home's attempts grows past period, which shows as a probe's lastProbeTime falling behind. Raise maxConcurrentRequests in your values file when that happens.

Troubleshooting¶

The controller rollout doesn't finish¶

With an unusable multi-cluster setting, the new controller pods exit when they start, and kubectl rollout status fails with deployment "ome-controller-manager" exceeded its progress deadline. Find the setting in the controller's log:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'invalid multi-cluster configuration'

The error names the setting. For example:

• A period without a unit gives routing.capacity.period: "10" is not a duration (use forms like "30s", "5m").
• A timeout as long as the period gives routing.capacity.timeout (10s) must be shorter than routing.capacity.period (10s).
• A period under the observer's floor gives routing.capacity.period (500ms) must be at least routing.observer.minPeriod (1s).
• An observer limit set to 0 or null gives routing.observer.maxSamples must be positive when routing is enabled.

Fix the value in your values file, and run helm upgrade again.

The TrafficMap doesn't pick up the poll¶

kubectl apply accepts some mistakes in spec.routing.capacity that OME refuses later. Until you fix them, OME stops polling and probing the service's homes, and its TrafficMap stops updating: it keeps its last weights, and Published turns False with the reason InvalidOwner. A new InferenceService gets no TrafficMap. A probe error has the same effect. Find the error in the controller's log:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep -E 'validate routing policy|resolve (capacity|probe) policy'

For example:

Fix the field, and apply isvc.yaml again.

A home falls back to its admitted replicas¶

When the CapacityFallback condition is True, its message counts the homes without a ceiling, as in 1 of 2 home(s) are using control-plane capacity; inspect spec.entries[].capacity.fallbackReason. List each home's reason:

kubectl get trafficmap qwen3-0-6b -n qwen3-0-6b \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{": "}{.capacity.fallbackReason}{"\n"}{end}'

When worker-b's poll reaches the model server instead of your reporter, and worker-a has its ceiling:

worker-a:
worker-b: capacity endpoint returned HTTP 404

The reason says what to fix:

Next steps¶

• Traffic map: how OME builds the TrafficMap and works out each home's weight.
• Configure routing health probes: set a cluster's weight to zero when its endpoint stops answering.
• Weight traffic for heterogeneous clusters: count one cluster's replicas for more than another's.
• Drain a workload cluster: hold a workload cluster at zero weight with kubectl ome traffic drain, whatever its capacity.
• Publish a global endpoint: serve the InferenceService on one hostname that splits its traffic by the TrafficMap's weights.