Weight traffic for heterogeneous clusters Preview Unreleased: coming in v1.3

A capacity factor gives a workload cluster whose replicas serve more requests a larger share of a multi-cluster InferenceService's traffic. By default, the service's TrafficMap counts every ready replica the same, which suits clusters that run the same accelerators. In the examples, a replica on worker-b serves three times the requests of one on worker-a, so worker-b gets a factor of 3 in spec.routing.capacityFactors.

Multi-cluster routing is alpha, still in development, and off by default. Its API, capacity factors included, can change without notice.

How factors change the weights¶

OME counts the replicas that can serve on a cluster: the lower of its admitted and its ready replicas. It multiplies the count by the cluster's factor, 1 unless you set one, and reduces the results across clusters to their smallest whole-number ratio.

For example, chat has four replicas on worker-a and two on worker-b, whose replicas each serve three times as many requests:

worker-b gets three of every five requests with a third of the replicas. Without the factor, the weights would be 2 and 1.

• A factor counts per replica, so a cluster's share still moves as it scales or as its replicas stop being ready.
• Only the ratios between factors matter: worker-a: "2" with worker-b: "6" gives the same weights as worker-b: "3" alone.
• Factors change only the weights. OME ignores them when it places the InferenceService, in mode Split too: to change where replicas run, change spec.placement.
• A capacity poll, a routing health probe or a drain can lower a cluster's weight, or hold it at 0, whatever its factor. See How OME sets the weights.

Step 1: Declare capacity factors¶

OME doesn't derive factors from the hardware: you choose them. Work out a cluster's factor as what one of its replicas serves divided by what one replica serves on your baseline cluster. If a worker-a replica sustains 20 requests per second at your latency target and a worker-b replica sustains 60, worker-b's factor is 3. A slower cluster gets a factor below 1, such as "0.5". Set factors only for the clusters that differ from the baseline.

Add spec.routing.capacityFactors to the InferenceService on the control-plane cluster. Its keys are WorkloadCluster names, as kubectl get workloadclusters lists them. Its values are positive, quoted Kubernetes quantities, such as "3", "1.5" or "500m". The control-plane cluster accepts 0 or a negative value, but OME then stops updating the TrafficMap, as The weights don't change shows.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat
  namespace: prod
spec:
  model:
    name: llama-3-1-8b-instruct
  runtime:
    name: srt-llama-3-1-8b-instruct
  engine:
    minReplicas: 2
    maxReplicas: 4
  placement:
    policy: ClusterAffinity
    mode: All
    clusterAffinity:
      - matchFields:
          - key: metadata.name
            operator: In
            values: [worker-a, worker-b]
  routing:
    capacityFactors:
      worker-b: "3"

In mode All, each workload cluster scales on its own, so worker-a can run four replicas while worker-b runs two.

kubectl apply -f chat.yaml

inferenceservice.ome.io/chat configured

OME rebuilds the TrafficMap when the InferenceService changes. Check that it has built the map from your change: the map's spec.observedISVCGeneration equals the InferenceService's metadata.generation.

kubectl get inferenceservice chat -n prod -o jsonpath='{.metadata.generation}{"\n"}'

6

kubectl get trafficmap chat -n prod -o jsonpath='{.spec.observedISVCGeneration}{"\n"}'

6

If the map's number stays lower, OME has stopped updating the map: see The weights don't change.

Step 2: Check the weights¶

List the entries' replica counts, factors and weights:

kubectl get trafficmap chat -n prod \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{" allocated="}{.capacity.allocated}{" ready="}{.capacity.ready}{" factor="}{.capacity.factor}{" weight="}{.weight}{"\n"}{end}'

worker-a allocated=4 ready=4 factor= weight=2
worker-b allocated=2 ready=2 factor=3 weight=3

• allocated and ready are the admitted and ready replicas. An empty value means 0.
• factor is the factor you set, in canonical form: "1.5" shows as 1500m. It's empty for worker-a, which counts as 1.
• weight is the cluster's share, relative to the other entries.

If a weight is 0, or isn't what you expect, compare the entry with Why a cluster's weight is zero.

If you publish a global endpoint, check that the publisher has written these weights with the map's staleness checks.

Move from the removed field¶

Older manifests set the factors in spec.placement.capacityFactors, which later became a deprecated alias of spec.routing.capacityFactors. The alias is gone: the InferenceService API no longer has a spec.placement.capacityFactors field. Move the map to spec.routing, with the same keys and values:

 spec:
   placement:
     policy: ClusterAffinity
     mode: All
     clusterAffinity:
       - matchFields:
           - key: metadata.name
             operator: In
             values: [worker-a, worker-b]
-    capacityFactors:
-      worker-b: "3"
+  routing:
+    capacityFactors:
+      worker-b: "3"

A manifest that keeps the old field sets no factors:

• By default, kubectl apply fails: kubectl asks the API server for strict validation, which rejects a field the schema doesn't have. See The API server rejects the InferenceService.
• A client that doesn't ask for strict validation gets the field dropped instead: the API server prunes spec.placement.capacityFactors as an unknown field and accepts the InferenceService, with at most a warning. Every cluster then counts as 1, and Step 2 shows an empty factor on every entry.

Troubleshooting¶

The weights don't change¶

After you apply a factor, the weights stay as they were, and the map's spec.observedISVCGeneration stays below the InferenceService's metadata.generation in Step 1's check. OME has stopped updating the map, for example because a factor in spec.routing.capacityFactors is 0 or negative.

With the kubectl ome plugin, kubectl ome placement explain chat -n prod shows Routing intent as Invalid when a factor is the cause. The command reads alpha features.

The error message shows only in the manager's log. Read every replica's log:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'validate routing policy'

The log shows an error such as validate routing policy: spec.routing.capacityFactors["worker-b"] must be positive, got "0". While it lasts, the TrafficMap and your gateway's route keep their last weights, and the map's Published condition turns False with reason InvalidOwner.

Set a positive factor, or remove the key, and apply the file again. OME then builds the map from the fixed spec. For the other errors that stop the map, see When the map stops updating.

A cluster's factor has no effect¶

Find the cluster's entry in Step 2:

The API server rejects the InferenceService¶

kubectl apply fails with an error that ends with unknown field "spec.placement.capacityFactors". The manifest sets the removed field, and kubectl's default strict validation has the API server reject a field the InferenceService schema doesn't have. Move the factors to spec.routing.capacityFactors, as Move from the removed field shows.

Clean up¶

To go back to plain replica counts, remove spec.routing.capacityFactors from chat.yaml, and apply the file again.

Next steps¶

• Traffic map: the inputs of a weight, and how to tell whether the table is current.
• Tune routing capacity polling: lower a cluster's count to the capacity that it reports.
• Configure routing health probes: take a cluster's traffic away when its endpoint fails.
• Drain a workload cluster: hold a cluster at weight 0 for maintenance.
• Publish a global endpoint: write the weights into your global gateway.