Traffic map Preview Unreleased: coming in v1.3

Placement decides which workload clusters run a multi-cluster InferenceService. Multi-cluster placement and routing are alpha, still in development, and off by default. With routing on, OME writes a TrafficMap that gives each of those clusters a weight. A publisher writes the weights into your global gateway, by default as a Gateway API HTTPRoute. Read the map to see why a cluster gets less traffic, or none, and whether the publisher has written the latest weights.

Turn on routing¶

Routing runs only on the control-plane cluster, and it's off by default. Turn it on in that cluster's ome-resources values:

ome:
  multicluster:
    enabled: true
    role: control-plane
    config:
      routing:
        enabled: true

helm upgrade restarts the manager to apply them. If you edit the inferenceservice-config ConfigMap instead, restart the manager yourself.

Routing then writes the maps. To publish them to your gateway, set the endpoint values in Publish a global endpoint. Until you do, Published is False with reason Withdrawn.

OME writes a map, with the same name and namespace, for each InferenceService that sets spec.placement. The obsolete annotations ome.io/accelerator-requirements and ome.io/cluster-selector don't opt a service in: OME rejects them on write, and a stored InferenceService that still carries one gets no map, only a validate placement intent: error in the manager's log. See When the map stops updating. To opt an InferenceService out, set spec.routing.enabled: false. Setting it to true doesn't turn routing on when the installation has it off.

OME deletes the map when the InferenceService stops being routed, for example because you turn routing off. When you delete the InferenceService, the finalizer ome.io/trafficmap-publisher holds the map until the publisher has removed the route. Placement keeps the InferenceService's copies on the workload clusters until the map is gone.

Read a TrafficMap¶

The examples use the InferenceService llama-chat in llama-demo, placed in mode Split on the workload clusters cluster-a and cluster-b.

To see a map, run kubectl get trafficmap, or its short names tm and tmap, on the control-plane cluster:

kubectl get trafficmap llama-chat -n llama-demo

NAME         MODE    PUBLISHED   ROUTABLE   OVERRIDE   REASON     AGE
llama-chat   Split   True        True       False      Routable   3h

MODE is the InferenceService's placement mode: Single runs it on one cluster, All on every cluster that admits it, and Split spreads its replicas across clusters. The other columns come from the map's conditions.

kubectl ome get trafficmaps lists the maps too, with their entry counts. -o wide adds the route and whether the publisher is current.

This is the map with four replicas ready on cluster-a and two on cluster-b, without its conditions and status.publisher:

apiVersion: ome.io/v1beta1
kind: TrafficMap
metadata:
  name: llama-chat
  namespace: llama-demo
  generation: 4
spec:
  service: llama-chat
  mode: Split
  observedISVCGeneration: 7
  entries:
    - cluster: cluster-a
      endpoint: http://llama-chat.llama-demo.cluster-a.example.com
      weight: 2
      healthy: true
      capacity:
        allocated: 4
        ready: 4
        source: ControlPlane
    - cluster: cluster-b
      endpoint: http://llama-chat.llama-demo.cluster-b.example.com
      weight: 1
      healthy: true
      capacity:
        allocated: 2
        ready: 2
        source: ControlPlane
status:
  sourceUID: 0f8e2c1a-5b7d-4e39-9c61-2d4a8b3f7e10
  published: true
  observedTrafficMapGeneration: 4
  gatewayRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llama-chat-global
    namespace: llama-demo

Each entry is a home of the InferenceService: a workload cluster that has admitted it and reports an endpoint for it. Condition reasons call the entries homes, as in AllHomesUnready. The entries hold these fields:

The map leaves out allocated and ready when they're 0. TrafficMapEntry in the API reference describes every field.

Treat the map as read-only: OME undoes a hand edit right away.

How OME sets the weights¶

OME works out an entry's weight in four steps:

1. It counts the replicas that can serve: the lower of capacity.allocated and capacity.ready, times the cluster's capacity factor, which is 1 unless you set one. For an InferenceService with an engine and a decoder, both counts are the lower of the two components' counts.
2. An entry that a routing health probe gates gets 0, with one exception that Routable describes.
3. It reduces the weights to their smallest whole-number ratio, so 4 and 2 become 2 and 1.
4. An entry that a drain holds gets 0. The other weights stay as they are.

Divide a weight by the sum of the weights for its share: in the example, cluster-a gets two thirds of the requests.

Four settings change the weights:

Why a cluster's weight is zero¶

Compare the entry with these rows, and take the first that matches:

If every entry has weight 0, or the table is empty, the Routable condition names the cause for the whole map.

Conditions¶

Routable, OverrideActive and CapacityFallback describe the table. Published says whether the publisher has written it to the gateway.

kubectl describe also shows PublicationFallback, which is False with reason Unsupported after the built-in publisher publishes successfully. It's Unknown while a publish is under way or failing. When Published has reason Withdrawn, Unpublished or Finalizing, PublicationFallback is False with one of those reasons too.

Routable¶

Routable says whether a gateway can send the InferenceService's requests anywhere:

OverrideActive¶

OverrideActive says whether drains hold any entry:

CapacityFallback¶

CapacityFallback is True when capacity polling is on and an entry has no usable report:

Published¶

The built-in publisher writes the table into an HTTPRoute, by default llama-chat-global in the InferenceService's namespace, and status.gatewayRef names it. The route has a backend for each entry, at the entry's weight, entries at weight 0 included. How publishing works lists what else it writes.

Published means that the publisher wrote the route. The HTTPRoute's own status says whether the gateway accepted it, as Check the publication shows.

When publishing fails, the route from the last success stays, and status.gatewayRef still names it. The condition's message is generic. The error is in the manager's log, on a line that reads TrafficMap publisher reconciliation failed. The publisher retries on its own.

Staleness checks¶

Before you rely on a map, check that it's current:

To compare the map's generation with the one that the publisher last published:

kubectl get trafficmap llama-chat -n llama-demo \
  -o jsonpath='{.metadata.generation} {.status.observedTrafficMapGeneration}{"\n"}'

4 4

The numbers match, so the publisher has published the latest table.

When the map stops updating¶

Some errors stop OME from updating a map. The map keeps its last table and conditions, and the route keeps its last published weights. When the error follows a spec change, Published turns False with reason InvalidOwner. The error shows only in the manager's log. Read every replica's log with kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1.

Next steps¶

• Publish a global endpoint: write the map's weights into your global gateway.
• Weight traffic for heterogeneous clusters: count one cluster's replicas for more with a capacity factor.
• Tune routing capacity polling: lower a cluster's weight to the capacity it reports.
• Configure routing health probes: gate a cluster's weight on an end-to-end probe.
• Drain a workload cluster: hold a cluster at weight 0 with kubectl ome traffic drain.