Traffic policy Unreleased: coming in v1.3

A traffic policy decides which of an InferenceService's pods serves each request. Use it to keep a conversation on one pod, to prefer the least busy pods, or to let the client pick the pod. You declare it in spec.traffic. OME writes it into your gateway's backend policy, an Envoy Gateway BackendTrafficPolicy or an Istio DestinationRule, and reports the result in status.traffic.

You set circuit breaking, retries and connection timeouts with traffic annotations, which OME writes into the same policy. A traffic policy doesn't split requests between revisions. During a canary, the new revision's share of requests follows its share of ready Instances, as Canary progression explains.

Where the policy takes effect¶

When the controller starts, it looks for Envoy Gateway's BackendTrafficPolicy CRD in gateway.envoyproxy.io/v1alpha1, then for Istio's DestinationRule CRD in networking.istio.io/v1. It writes policies for the first gateway it finds:

That Service leads to the router, or to the engine when there's no router. On a cluster with neither CRD, status.traffic reports NoTranslatorAvailable.

The controller logs its choice as Selected traffic translator. It picks only when it starts, so restart it after you install or remove a gateway's CRDs. With OME in the ome namespace:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

After a switch from Istio to Envoy Gateway, the DestinationRules that OME wrote keep applying until you delete them, for example with kubectl delete destinationrule llama-chat -n llama-demo.

Set a traffic policy¶

The examples use the llama-demo namespace and the pre-configured model llama-3-2-1b-instruct, as in A minimal InferenceService. This InferenceService runs two engine pods, and keeps each chat session on one of them:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 2
    maxReplicas: 2
  traffic:
    algorithm: ConsistentHash
    consistentHash:
      type: Header
      headers:
        - name: x-session-id

Apply the file:

kubectl apply -f llama-chat.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-chat created

• algorithm: ConsistentHash sends the requests that share the value of a request attribute to the same pod.
• consistentHash makes that attribute the x-session-id header. Every request with the same session ID reaches the same pod, as long as the set of pods doesn't change.

OME acts on traffic intent: any field in spec.traffic, or any circuit breaker, retry, timeout or pass-through annotation. Without intent, the defaults apply.

OME names the policy after the InferenceService, creates it in the same namespace, and owns it. It overwrites changes you make to the policy by hand, and deletes the policy when you remove all traffic intent. Kubernetes deletes the policy with the InferenceService. A change to spec.traffic during a rollout can wait until the rollout finishes.

One policy covers the whole InferenceService, and spec.traffic has no per-component settings. To give one component other settings, write a policy of your own under another name. Under Istio, that's a DestinationRule for the component's Service, such as llama-chat-engine. Under Envoy Gateway, your BackendTrafficPolicy targets a route that OME's policy also targets, and Envoy Gateway's rules for overlapping policies decide which settings apply. Unlike the default policy, the policy that intent writes doesn't merge with a BackendTrafficPolicy that targets the Gateway.

Read the traffic status¶

status.traffic shows what OME did with your intent. On Envoy Gateway, after OME writes the policy for llama-chat.yaml:

kubectl get inferenceservice llama-chat -n llama-demo -o custom-columns='ALGORITHM:.status.traffic.algorithm,POLICY:.status.traffic.backendPolicyResource.kind,READY:.status.traffic.conditions[?(@.type=="BackendPolicyReady")].status,REASON:.status.traffic.conditions[?(@.type=="BackendPolicyReady")].reason'

ALGORITHM        POLICY                 READY     REASON
ConsistentHash   BackendTrafficPolicy   Unknown   Pending

The BackendPolicyReady condition is one of:

On Envoy Gateway, BackendPolicyReady stays Pending even after the gateway accepts the policy. To see the gateway's verdict, read the conditions under status.ancestors in the policy, for example with kubectl get backendtrafficpolicy llama-chat -n llama-demo -o yaml. This is a known bug. On Istio, the condition follows the DestinationRule's Reconciled condition, and stays Pending until Istio sets one.

When the gateway can't apply a field or annotation that you declared, OME leaves it out of the policy and lists it in a BackendPolicyUnsupportedFields condition. Under Istio, that happens to endpoint override and to some traffic annotations.

Choose a load-balancing algorithm¶

spec.traffic.algorithm picks how the gateway chooses a pod for each request. Every value works on both Envoy Gateway and Istio:

For example:

spec:
  traffic:
    algorithm: LeastRequest

Without algorithm, the gateway uses its own default, LeastRequest on Envoy Gateway, and status.traffic.algorithm shows Default.

Session affinity with ConsistentHash¶

With algorithm: ConsistentHash, the gateway hashes a request attribute and sends the requests with the same hash to the same pod. Hashing on a session ID keeps a conversation on the engine that has already cached its prompt prefix. Adding or removing a pod moves some sessions to other pods. consistentHash.type picks the attribute:

spec:
  traffic:
    algorithm: ConsistentHash
    consistentHash:
      type: Header
      headers:
        - name: x-tenant-id
        - name: x-session-id

spec:
  traffic:
    algorithm: ConsistentHash
    consistentHash:
      type: Cookie
      cookie:
        name: session-affinity
        ttlSeconds: 3600

spec:
  traffic:
    algorithm: ConsistentHash
    consistentHash:
      type: SourceIP

Endpoint override¶

endpointOverride lets the client pick the pod, for example when an external scheduler has already chosen one. With this setting, a request that carries an x-endpoint-hostport header goes to the pod that the header names:

spec:
  traffic:
    algorithm: LeastRequest
    endpointOverride:
      type: Header
      headers:
        - name: x-endpoint-hostport

The header's value is the pod's address and port, such as 10.0.0.5:8080. The vLLM runtime in these examples listens on port 8080. When the header is missing or names an unhealthy pod, the gateway falls back to algorithm, so always declare one.

Only Envoy Gateway supports endpoint override. Under Istio, the BackendPolicyUnsupportedFields condition names it:

translator "istio" does not honor 1 typed spec.traffic field(s) [spec.traffic.endpointOverride.type=Header]

Defaults when you declare nothing¶

Without traffic intent, the gateway uses its own defaults, and the InferenceService has no status.traffic.

The exception is ingress through the Gateway API. There, OME writes a default BackendTrafficPolicy, which needs Envoy Gateway v1.7 or later. It hashes on the x-routing-key and X-SMG-Routing-Key headers. To hash on others, set consistentHashHeaders, or the ome.io/ingress-consistent-hash-headers annotation on one InferenceService. That annotation changes the default policy instead of replacing it.

The default policy covers only the top-level route, the HTTPRoute named after the InferenceService. A request through a component's own route, such as /llama-demo/llama-chat-engine/, isn't hashed, and the gateway balances it with its own default. To hash on a component's route too, declare the headers in spec.traffic, as in Session affinity with ConsistentHash: the policy that intent writes targets all of the InferenceService's routes.

The default policy also merges. It sets mergeType: StrategicMerge, so when a BackendTrafficPolicy of your own targets the Gateway, the top-level route inherits your settings, such as circuit breakers and timeouts, and the default policy adds the hashing on top.

Traffic intent replaces the default policy entirely, and removing the intent brings it back. So a traffic annotation on its own, such as ome.io/circuit-breaker-max-connections, ends the header hashing. To keep it, also declare algorithm: ConsistentHash with those headers. The policy that intent writes sets no merge type, so on the routes it targets, it overrides a Gateway-level policy instead of inheriting its settings. Declare the circuit breakers and timeouts you still need with traffic annotations.

Troubleshooting¶

The webhook rejects the InferenceService¶

kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, then a message that ends in a reason in parentheses:

spec.traffic.consistentHash is required when spec.traffic.algorithm=ConsistentHash (MissingConsistentHashSpec)

Traffic annotations lists the checks on annotations.

The traffic status doesn't change¶

When OME can't write the policy, it records a TrafficReconcileError event on the InferenceService and retries. That happens when another object has the policy's name, or when the API server rejects the policy. Until the error clears, the InferenceService's whole status, status.traffic included, keeps its earlier values. Read the events:

kubectl get events -n llama-demo --field-selector reason=TrafficReconcileError -o custom-columns='OBJECT:.involvedObject.name,MESSAGE:.message'

OBJECT       MESSAGE
llama-chat   backend policy with the OME-managed name exists but is not owned by this InferenceService: llama-demo/llama-chat already exists with no controller owner reference (hand-authored)

Here, a policy that someone wrote by hand already has the name llama-chat. Delete it, or re-create it under another name, and OME writes its policy when it retries.

Next steps¶

• Ingress and external access: the routes and the Service that the policy applies to.
• Configure ingress: turn on Gateway API ingress with Envoy Gateway.
• Traffic annotations: circuit breaking, retries, timeouts and pass-through fields.
• kubectl ome traffic: show the traffic status that OME reports for an InferenceService, and compare it with what you declared.