Traffic annotations Unreleased: coming in v1.3

Annotations on an InferenceService set circuit breakers, retries and connection timeouts for its backends, so you can cap the load on its pods, retry failed requests and close idle connections. OME writes them into one backend policy for your gateway: a gateway.envoyproxy.io/v1alpha1 BackendTrafficPolicy on Envoy Gateway, or a networking.istio.io/v1 DestinationRule on Istio. Pass-through annotations set the policy's other fields, and ome.io/service-type sets the type of the Services that OME creates. For load balancing and session affinity, use spec.traffic: see Traffic policy.

OME writes the policy while the InferenceService has traffic intent: spec.traffic, or a circuit breaker, retry, timeout or pass-through annotation. It deletes the policy when you remove them all. ome.io/service-type and ome.io/load-balancer-ip set only Services, so they aren't traffic intent. The policy takes effect on Envoy Gateway when ingress uses the Gateway API, and on Istio while ingress creation is off, which is the default. The controller picks the gateway when it starts, and prefers Envoy Gateway when both are installed: see Where the policy takes effect.

Annotation values are strings, so quote numbers in YAML: "1024", not 1024. Retry and timeout durations take the units ns, us, ms, s, m and h, as in 10s or 1m30s. OME writes them in normal form, so 5m becomes 5m0s. When you leave out a circuit breaker, retry or timeout annotation, the gateway's default applies. The circuit breaker, retry, timeout and pass-through annotations need v1.3, while ome.io/service-type and ome.io/load-balancer-ip work on v1.2.2 too.

Circuit breaker annotations¶

Circuit breakers cap the load that the gateway sends to the InferenceService's pods. The values are whole numbers. For what the fields do and their defaults, see Envoy Gateway's BackendTrafficPolicy or Istio's DestinationRule.

With spec.traffic.algorithm: ConsistentHash, the webhook warns about ome.io/circuit-breaker-per-endpoint-max-connections above 1.

Retry annotations¶

Retries resend a request that failed. OME writes them only for Envoy Gateway. On Istio, it lists them in BackendPolicyUnsupportedFields.

OME can't set spec.retry.retryOn.httpStatusCodes, the status codes that retriable-status-codes retries on: ome.io/retry-on takes only condition names, and pass-throughs set only single values.

Timeout annotations¶

These annotations set connection timeouts. OME sets the request timeout on its HTTPRoutes from each component's timeoutSeconds or a cluster-wide default: see Configure route timeouts. For a per-try timeout, use ome.io/retry-per-try-timeout.

Example¶

This InferenceService uses the BaseModel and runtime from Serve models from a PVC, in the namespace llama-demo.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
  annotations:
    ome.io/circuit-breaker-max-connections: "1024"
    ome.io/circuit-breaker-max-pending-requests: "256"
    ome.io/retry-attempts: "3"
    ome.io/retry-on: "connect-failure,reset,5xx"
    ome.io/retry-per-try-timeout: "30s"
    ome.io/timeout-idle: "5m"
    ome.io/timeout-tcp-connect: "10s"
spec:
  model:
    name: llama-3-2-1b-instruct
    kind: BaseModel
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

kubectl apply -f llama-chat.yaml

inferenceservice.ome.io/llama-chat created

What OME writes depends on the gateway:

spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llama-chat
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: llama-chat-engine
  circuitBreaker:
    maxConnections: 1024
    maxPendingRequests: 256
  retry:
    numRetries: 3
    perRetry:
      timeout: 30s
    retryOn:
      triggers:
      - connect-failure
      - reset
      - 5xx
  timeout:
    http:
      connectionIdleTimeout: 5m0s
    tcp:
      connectTimeout: 10s

kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.traffic.backendPolicyResource.kind}'

BackendTrafficPolicy

spec:
  host: llama-chat.llama-demo.svc.cluster.local
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1024
        connectTimeout: 10s
      http:
        http1MaxPendingRequests: 256
        idleTimeout: 5m0s

kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.traffic.backendPolicyResource.kind}'

DestinationRule

kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.traffic.conditions[?(@.type=="BackendPolicyReady")].reason}'

NoTranslatorAvailable

Pass-through prefixes¶

A pass-through annotation sets any field of the policy. Its key is a prefix, then the field's path, with dots between its parts:

OME converts the value to a whole number, a boolean or a decimal, trying them in that order, and keeps it as a string when none fits. So a pass-through can't set a list, an object, or a string that looks like a number or a boolean. OME applies pass-throughs last, so they win over any annotation or spec.traffic field that sets the same field.

metadata:
  annotations:
    ome.io/retry-attempts: "3"
    ome.io/retry-on: "connect-failure"
    ome.io/btp.retry.numRetries: "5"

metadata:
  annotations:
    ome.io/dr.outlierDetection.consecutive5xxErrors: "5"
    ome.io/dr.outlierDetection.baseEjectionTime: "30s"

The webhook rejects the other gateway's prefix, but doesn't check paths or values. When a value has the wrong type, the API server rejects the policy, and OME records a TrafficReconcileError event. A misspelled path is dropped. To check what OME wrote, read the policy, for example with kubectl get backendtrafficpolicy llama-chat -n llama-demo -o yaml.

Status and events¶

OME reports the policy it wrote in status.traffic, with a BackendPolicyReady condition for whether the gateway accepted it: see Read the traffic status. On Envoy Gateway, the condition stays Pending even after the gateway accepts the policy. To see the gateway's verdict, read the conditions under status.ancestors in the BackendTrafficPolicy. This is a known bug.

When OME can't write the policy, it records a TrafficReconcileError event, and the InferenceService's status stops changing until the error clears. The traffic status doesn't change lists the causes and how to fix them.

Unsupported annotations¶

On Istio, OME leaves out the retry annotations and the ones marked Not supported, and writes the rest. It lists what it left out in the condition BackendPolicyUnsupportedFields. With the three retry annotations:

- type: BackendPolicyUnsupportedFields
  status: "True"
  reason: UnsupportedField
  message: 'translator "istio" does not honor 3 operator-declared annotation(s) [ome.io/retry-attempts ome.io/retry-on ome.io/retry-per-try-timeout]'

On Istio, the message also lists endpoint override when you set it.

Admission validation¶

The admission webhook checks the circuit breaker, retry and timeout annotations, and pass-through prefixes, when you create or update an InferenceService. When it rejects one, kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, then a message that ends in a reason in parentheses:

annotation "ome.io/retry-attempts" set to 3 requires "ome.io/retry-on" to specify retry conditions (MissingRetryOn)

With spec.traffic.algorithm: ConsistentHash, the webhook admits ome.io/circuit-breaker-per-endpoint-max-connections above 1, and kubectl prints this after Warning::

annotation "ome.io/circuit-breaker-per-endpoint-max-connections"=4 combined with traffic.algorithm=ConsistentHash usually defeats sticky routing; set to 1 to keep sessions on a single pod

Service type annotation¶

ome.io/service-type sets the type of the Services that OME creates for an InferenceService: ClusterIP, the default, NodePort or LoadBalancer. Values are case-sensitive, and any other value gives ClusterIP.

metadata:
  annotations:
    ome.io/service-type: LoadBalancer

The type applies to the Service named after the InferenceService, which exists only while ingress creation is off. OME copies the InferenceService's annotations that start with service.beta.kubernetes.io/, cloud.google.com/ or service.kubernetes.io/ to that Service. On a NodePort or LoadBalancer Service, OME resets changes that you make, such as setting externalTrafficPolicy. Give Services of your own other names: OME changes or deletes any Service with the InferenceService's name.

The type also applies to each component Service, such as llama-chat-engine, when OME creates it, so with LoadBalancer each component can get a load balancer of its own. A later change doesn't reach an existing component Service. ome.io/load-balancer-ip sets spec.loadBalancerIP on component Services of type LoadBalancer, not on the Service named after the InferenceService. Expose a service without ingress shows how to use ome.io/service-type, and how to keep the component Services internal.

Related pages¶

• Traffic policy: the typed spec.traffic fields, and where the policy takes effect.
• kubectl ome traffic: show the policy that OME wrote and its status.
• Configure route timeouts: the request timeout on OME's HTTPRoutes.
• Expose a service without ingress: expose an InferenceService with ome.io/service-type.
• Configure ingress: turn on ingress creation.
• Labels and annotations: the other labels and annotations OME reads and sets.
• OME API: the status.traffic fields.