Ingress and external access

With ingress creation on, OME exposes each InferenceService outside the cluster through a Kubernetes Ingress or Gateway API HTTPRoutes, and reports the address that clients use in status.url. Ingress creation is off by default. Until you turn it on, OME puts a Service in front of the model, and you decide how to expose it.

You set ingress with the ome-resources chart values under ome.controller.ingressGateway, which the chart writes to the inferenceservice-config ConfigMap in the ome namespace. Configure ingress walks through the setup and lists every setting. The examples use an InferenceService named llama-chat in the namespace llama-demo.

What OME creates for external access¶

Two of the values decide what OME creates for each InferenceService:

The annotation ome.io/ingress-disable-creation overrides disableIngressCreation for one InferenceService: "true" turns ingress creation off, and any other value turns it on. enableGatewayAPI has no annotation, so it applies to every InferenceService. Per-service overrides lists the annotations for the other settings.

OME copies the InferenceService's annotations to its Ingress or HTTPRoutes, so annotations for your ingress controller or Gateway go on the InferenceService. OME updates a route's annotations only when the route's spec changes too.

HTTPRoutes need v1.3 or later. On v1.2.2, enableGatewayAPI: true makes the ingress step fail for every InferenceService, so use an Ingress.

Turning ingress creation on deletes the external Service, so move clients in the cluster to a route, or to a component's Service such as llama-chat-engine. Turning it off, or switching between an Ingress and HTTPRoutes, leaves the old Ingress or HTTPRoutes serving traffic until you delete them, as Turn off ingress creation shows.

The Ingress and HTTPRoutes send requests to the Services that OME creates for the InferenceService's components: <name>-engine, <name>-decoder and <name>-router. So they work the same for OMENative components as for RawDeployment ones. They also work the same when the InferenceService fronts standalone replicas with spec.replicaRefs; the Services then select the replicas' pods, as Referenced replicas describes. Keep InferenceService names to 56 characters, or 55 with a decoder. With longer names, the routes point at Services that don't exist.

The external Service¶

With ingress creation off, OME puts a ClusterIP Service named after the InferenceService in front of its router, or its engine when there's no router. Cluster-local InferenceServices don't get one. The Service listens on the runtime's port, 8080 for llama-chat.

To reach the model from outside the cluster, set the annotation ome.io/service-type to LoadBalancer or NodePort, or put your own Ingress or Gateway in front of the Service. Expose a service without ingress walks through it.

OME treats any Service with the InferenceService's name as its own, and rewrites or deletes it, so give your own Services other names.

Through a Kubernetes Ingress¶

OME creates one Ingress, named after the InferenceService, with the IngressClass in ome.controller.ingressGateway.ingressGateway.className. The chart sets it to istio, so change it to your ingress controller's class, such as nginx. Each rule matches the path prefix / on a hostname of its own.

Hostnames¶

With domain: example.com, the Ingress for llama-chat has these hostnames:

The hostnames come from domainTemplate, a Go template that defaults to {{ .Name }}.{{ .Namespace }}.{{ .IngressDomain }}. In the template, .Name is the InferenceService's name for the top-level hostname, and the component's Service name for the others.

Through the Gateway APIUnreleased: coming in v1.3¶

To create HTTPRoutes, set these values in the ome-resources chart:

ome:
  controller:
    ingressGateway:
      disableIngressCreation: false
      enableGatewayAPI: true
      domain: example.com
      omeIngressGateway: envoy-gateway-system/internal-gateway

• domain is the domain that the hostnames end in. The chart sets svc.cluster.local, so set a domain that your clients can resolve.
• omeIngressGateway is the Gateway that the HTTPRoutes attach to, as <namespace>/<name>. The chart leaves it empty, and the API server rejects HTTPRoutes without it.

Turning on enableGatewayAPI takes a controller restart. helm upgrade restarts the controller when the values change. If you edit the ConfigMap directly instead, restart it yourself:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

HTTPRoutes also need:

• a Gateway listener whose allowedRoutes.namespaces.from is All, or a Selector that matches the InferenceService's namespace. The default, Same, admits only routes in the Gateway's namespace;
• the Gateway API CRDs. If you install them after the controller starts, restart it, or IngressReady lags behind the Gateway;
• Envoy Gateway v1.7 or later, for the default BackendTrafficPolicy that OME writes for an InferenceService without traffic intent.

If OME can't write a route or the policy, it still creates the workloads, but the InferenceService's status stops updating. Troubleshooting shows how to find the cause in the controller's log.

Routes¶

For llama-chat, OME creates up to four HTTPRoutes, all with the hostname llm.example.com in the default scheme:

Clients use the top-level route, llama-chat. The other routes each reach one component, bypassing the router. OME creates a component's route once the component is ready. For an OMENative component, a serving Instance is enough: OME creates and updates the route even while the component's EngineReady, RouterReady or DecoderReady condition is False, as long as the component's lifecycle reports readyReplicas and servingReplicas above 0. So a component whose condition turns False with the reason InsufficientAvailable keeps its route, and IngressReady can stay True, while OME restores the Instances.

Each route rewrites its path prefix to /, so a request for /llama-demo/llama-chat/v1/models reaches the model server as /v1/models. It also sets the headers OMe-Isvc-Name and OME-Isvc-Namespace to the InferenceService's name and namespace.

A route's request timeout is its component's timeoutSeconds, such as spec.engine.timeoutSeconds, or else defaultRouteTimeoutSeconds. An InferenceService with spec.replicaRefs has no component blocks, so all of its routes take the default: see Referenced replicas. The chart sets defaultRouteTimeoutSeconds to 0, which turns the timeout off. With null, the Gateway's default applies, and Envoy's 15 seconds cuts long generations short. timeoutSeconds applies only to HTTPRoutes. Configure route timeouts has examples.

Hostnames¶

OME builds the hostnames in one of two schemes:

• In the shared scheme, the default, every HTTPRoute has the hostname <sharedHostPrefix>.<domain>, which is llm.example.com with the chart's sharedHostPrefix: llm, and a path prefix of its own.
• In the per-service scheme, which perISVCSubdomain: true turns on, an InferenceService's HTTPRoutes use a hostname from domainTemplate, such as llama-chat.llama-demo.example.com, and match the path prefix / with no rewrite.

additionalIngressGateways attaches the routes to more Gateways, each with its own domain, and namespaceIngressGateways picks the Gateways for the InferenceServices in a namespace. See Use multiple gateways and Use per-namespace gateways.

Send requests¶

With the values in Through the Gateway API, create llama-chat. It serves the pre-configured model llama-3-2-1b-instruct, as A minimal InferenceService does:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine: {}

Apply the file:

kubectl apply -f llama-chat.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-chat created

Once the engine is ready, OME creates two HTTPRoutes: llama-chat, which clients use, and llama-chat-engine. It also writes a default BackendTrafficPolicy named llama-chat. List the routes:

kubectl get httproutes -n llama-demo

NAME                HOSTNAMES             AGE
llama-chat          ["llm.example.com"]   4m
llama-chat-engine   ["llm.example.com"]   4m

Read the address from the status:

kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.url}'

http://llm.example.com/llama-demo/llama-chat/

llm.example.com must resolve to your Gateway's address, through DNS or an entry in /etc/hosts. List the models that the InferenceService serves:

curl -s http://llm.example.com/llama-demo/llama-chat/v1/models | jq -r '.data[].id'

vllm-model

The runtime serves the model as vllm-model, which it sets with --served-model-name. Send chat completions to http://llm.example.com/llama-demo/llama-chat/v1/chat/completions, with "model": "vllm-model" in the body.

Status¶

For llama-chat, with domain: example.com and the chart's other values, status.url is:

• With an Ingress, the port is the router's or the engine's Service port. Send requests to your ingress controller's port instead.
• With HTTPRoutes, status.url is the address on the primary Gateway. With more Gateways, status.addresses lists the address on each, as Use multiple gateways shows.
• With HTTPRoutes, send requests to status.url. The routes don't serve the component URLs in status.components.

When IngressReady is True¶

The InferenceService is Ready only when IngressReady is True, as Status describes:

When IngressReady is False, its reason and message say why:

The messages name the component, the engine in these examples. Troubleshooting in Configure ingress says what to do.

Route edits and gateway acceptance¶

A Gateway's acceptance counts only for the route generation it observed. An HTTPRoute is ready when every Gateway in its spec.parentRefs reports Accepted: True at the route's current generation, with no False condition at that generation. A Gateway that's missing from the route's status.parents, or whose conditions describe an older generation, reads as not accepted, so IngressReady is False with the reason ParentStatusNotAvailable.

So a change that rewrites a route's spec — a component's timeoutSeconds, a switch of host scheme, or a change to the Gateways the routes attach to — drops IngressReady to False the moment OME updates the route, with the reason ParentStatusNotAvailable and the message engine HTTPRoute awaiting gateway reprogramming, naming the component whose route changed. The InferenceService's Ready condition drops with it. This is the normal reprogramming window, not a failure: the update gave the route a new generation, so the acceptance the Gateways reported for the previous generation no longer counts. While the wait goes on, the message becomes Engine HttpRouteNotReady, with the same reason. Both conditions turn True again once every Gateway accepts the new generation. If ParentStatusNotAvailable persists, a Gateway isn't reporting on the route: see Troubleshooting.

Referenced replicas¶

An InferenceService that fronts standalone InferenceReplicas gets the same external access as one that declares its components. A role that spec.replicaRefs names, as Referenced replicas on the InferenceServices page describes, counts as its component in every choice on this page: whether requests go to the router or the engine, the "with a router" and "with a decoder" rows of the hostname and route tables, and status.url. For llama-fronted, which fronts the standalone engine replica llama-pool, status.url with ingress creation off is http://llama-fronted.llama-demo.svc.cluster.local:8080, and otherwise the address in the Status table, with llama-fronted in place of llama-chat.

What changes is whose pods serve the requests. A standalone replica's pods are named and labeled for the replica, not for the service that fronts it:

• The Ingress and the HTTPRoutes send requests to the component Services, llama-fronted-engine here. Such a Service selects its replica's pods — llama-pool-engine-0-default-0 and so on — by the label ome.io/inferenceservice: llama-pool. When the replica runs each Instance as a leader and workers, the Service selects only the leader pods.
• The external Service llama-fronted also selects the replica's pods: the referenced router's, or the referenced engine's when there's no router.

A component Service takes its ports from the replica's runners — the leader's when it has one — or, when the replica carries no runners, from the replica's current revision. Until the replica reports a revision, the Service publishes the default port, 8080.

A referenced role backs its route by the same rules as a declared component. The Ingress waits for the role it targets to be ready, and a referenced role's ready condition follows the replica's counts. For HTTPRoutes, the Routes relaxation applies as written: a role whose ready condition turns False gets and keeps its route while the replica reports readyReplicas and servingReplicas above 0. When the named replica is missing or invalid, the role's condition turns False with the reason ReplicaRefMissing or ReplicaRefInvalid, and the role stops backing its route: OME stops creating and updating the role's routes, and IngressReady turns False with the reason ComponentNotReady.

The routes' request timeout comes from defaultRouteTimeoutSeconds alone. The webhook rejects spec.engine, spec.router and spec.decoder next to spec.replicaRefs, and an InferenceReplica carries no timeoutSeconds, so no per-service setting exists: every route of a fronting service takes the default or, when it's unset, has no timeout of its own, and the gateway's default applies. Referenced replicas on Configure route timeouts shows how to set the default.

Security¶

OME doesn't set up TLS. Terminate it at your Gateway's listeners or in your ingress controller. The urlScheme value and the annotation ome.io/ingress-url-scheme change only the scheme that OME reports. Set https when you terminate TLS, so that status.url matches.

Nothing authenticates requests by default. To control who can send them, use your Gateway's or ingress controller's authentication.

Cluster-local services¶

The label networking.knative.dev/visibility: cluster-local stops OME from creating the external Service, and makes OME delete an existing one. The label doesn't stop an Ingress or HTTPRoutes. To keep an InferenceService inside the cluster whatever the config says, add the label and the annotation ome.io/ingress-disable-creation: "true". OME then sets no status.url, and clients in the cluster use the router's or the engine's Service, such as http://llama-chat-engine.llama-demo.svc.cluster.local:8080.

On v1.2.2, while ingress creation is off, a cluster-local InferenceService never reports Ready. Every reconcile fails after OME creates or updates its serving workload, and OME records an InternalError warning event. This is a known bug. With ingress creation off, the external Service is a ClusterIP Service, so on v1.2.2, leave the label off: the model stays inside the cluster without it.

Next steps¶

• Configure ingress: turn on an Ingress or HTTPRoutes, and set the domain, the Gateway and the other settings.
• Expose a service without ingress: make the external Service a LoadBalancer or NodePort Service.
• Choose a Gateway API host scheme: pick the shared or the per-service scheme.
• Configure route timeouts: keep long generations from timing out.
• Traffic policy: set load balancing and session affinity at the gateway.