Use per-namespace gateways Unreleased: coming in v1.3

Route each namespace's InferenceServices through that namespace's own Gateway, and override the gateway for a single service with annotations.

Give the InferenceServices in one namespace their own Gateway and domain, so that their listeners, TLS certificate and DNS stay separate from the rest of the cluster. You map the namespace prod to the Gateway prod-gw in the Helm values, check its routes, and then override the Gateways of one InferenceService with an annotation. By default, every InferenceService's HTTPRoutes attach to the cluster's Gateway.

Before you begin¶

OME v1.3 or later, installed with the ome-resources chart in the namespace ome, and kubectl access to the cluster. See Install OME.
Ingress creation turned on with the Gateway API. It's off by default. See Configure ingress.
An InferenceService in prod whose engine is ready. The examples use llama-chat, which has no router.
Envoy Gateway, with two Gateways in the namespace envoy-gateway-system: shared-gw, the cluster's Gateway, for the domain example.com, and prod-gw for prod.example.com.
A DNS record that sends llm.prod.example.com to prod-gw. kubectl get gateway prod-gw -n envoy-gateway-system shows the Gateway's address.
A listener on prod-gw that allows routes from prod. By default, a listener allows routes only from its own Gateway's namespace. This listener does, for any host in prod.example.com:

YAML

listeners:
  - name: http
    protocol: HTTP
    port: 80
    hostname: "*.prod.example.com"
    allowedRoutes:
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: prod

Step 1: Map `prod` to its Gateway¶

Add an entry for prod to your ome-resources values file. With Gateway API routing from Configure ingress, the ingress values look like this:

values.yaml

ome:
  controller:
    ingressGateway:
      disableIngressCreation: false
      enableGatewayAPI: true
      domain: example.com
      omeIngressGateway: envoy-gateway-system/shared-gw
      namespaceIngressGateways:
        prod:
          primary:
            omeIngressGateway: envoy-gateway-system/prod-gw
            ingressDomain: prod.example.com

namespaceIngressGateways maps a namespace to an entry for its InferenceServices. primary and each Gateway in additional take the last three fields:

Key	What it sets
The key, here `prod`	The InferenceServices' namespace, not the Gateway's.
`primary`	Replaces the cluster-wide `omeIngressGateway`, `domain` and `omeIngressGatewayClass` together, when it sets `omeIngressGateway`. A field you leave out stays empty, even if the cluster-wide value is set.
`additional`	Optional. More Gateways, after the primary, in place of the cluster-wide `additionalIngressGateways`. Set it to `[]` for none, or leave it out to keep the cluster-wide list.
`omeIngressGateway`	The Gateway, as `namespace/name`.
`ingressDomain`	The domain of the host on this Gateway. Always set it.
`class`	Optional. Names this Gateway's address in `status.addresses`, such as `internal`. It must be a lowercase RFC 1123 label.

Give the namespace a domain of its own, so that DNS can send its host to prod-gw. With the default shared host, the InferenceServices in prod share the host llm.prod.example.com, and those in other namespaces stay on shared-gw at llm.example.com. With per-service subdomains and the default domainTemplate, the host already names the namespace, so the entry can reuse the cluster-wide domain: ingressDomain: example.com gives llama-chat.prod.example.com.

For HTTPS, add a certificate for *.prod.example.com to an HTTPS listener on prod-gw. The URLs in status take their scheme from the cluster-wide urlScheme, or from an InferenceService's ome.io/ingress-url-scheme annotation.

Upgrade at the installed chart version, so that OME doesn't move ahead of the ome-crd chart. The CHART column of helm list -n ome shows it, here ome-resources-1.3.0:

Shell

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm prints Release "ome" has been upgraded. Happy Helming! and the new revision. The upgrade restarts the controller, which then applies the entry to the InferenceServices in prod.

Check the entry that the controller reads:

Shell

kubectl get configmap inferenceservice-config -n ome \
  -o jsonpath='{.data.ingress}' | jq .namespaceIngressGateways

Output

{
  "prod": {
    "primary": {
      "ingressDomain": "prod.example.com",
      "omeIngressGateway": "envoy-gateway-system/prod-gw"
    }
  }
}

Step 2: Check the routes¶

List the HTTPRoutes in prod with their Gateways and hosts:

Shell

kubectl get httproute -n prod \
  -o 'custom-columns=NAME:.metadata.name,GATEWAY:.spec.parentRefs[*].name,HOSTNAME:.spec.hostnames[*]'

Output

NAME                GATEWAY   HOSTNAME
llama-chat          prod-gw   llm.prod.example.com
llama-chat-engine   prod-gw   llm.prod.example.com

Both routes now name prod-gw, at the new host. An InferenceService with a router or a decoder has a route for that component too.

Check that the InferenceService's URL uses the new host, and that IngressReady is True. If it stays False, see IngressReady is False.

Shell

kubectl get inferenceservice llama-chat -n prod \
  -o 'custom-columns=URL:.status.url,INGRESS:.status.conditions[?(@.type=="IngressReady")].status'

Output

URL                                            INGRESS
http://llm.prod.example.com/prod/llama-chat/   True

To send a request, use the Gateway API request in Configure ingress, with the host llm.prod.example.com and the path /prod/llama-chat/.

Override the Gateways for one InferenceService¶

Two annotations on an InferenceService override the entry for its namespace. OME picks the primary Gateway and the additional Gateways separately, each from the first row that sets it:

Source	Primary Gateway	Additional Gateways
The InferenceService's annotations	`ome.io/ingress-gateway`	`ome.io/ingress-additional-gateways`
The entry for its namespace	`primary`, when it sets `omeIngressGateway`	`additional`, when set
The cluster-wide values	`omeIngressGateway`	`additionalIngressGateways`

So an InferenceService in prod with only ome.io/ingress-additional-gateways keeps prod-gw as its primary. An empty annotation still skips its part of the entry, and so does an ome.io/ingress-additional-gateways value that isn't a JSON list. The cluster-wide value applies instead. status.url is the address on the primary Gateway, and status.addresses lists one for each Gateway, named by its class.

ome.io/ingress-gateway replaces only the Gateway. The host on it uses the cluster-wide domain, or the InferenceService's ome.io/ingress-domain, and its address takes the cluster-wide omeIngressGatewayClass, not the entry's class. Override the gateways for one service describes the annotations' values.

Say a Gateway partner-gw serves partner.example.com, and its listener allows routes from prod. To serve llama-chat through it too, at llm.partner.example.com:

Shell

kubectl annotate inferenceservice llama-chat -n prod \
  ome.io/ingress-additional-gateways='[{"omeIngressGateway":"envoy-gateway-system/partner-gw","ingressDomain":"partner.example.com"}]'

Output

inferenceservice.ome.io/llama-chat annotated

Check that the route now attaches to both Gateways, the primary first:

Shell

kubectl get httproute llama-chat -n prod \
  -o 'custom-columns=GATEWAY:.spec.parentRefs[*].name,HOSTNAME:.spec.hostnames[*]'

Output

GATEWAY              HOSTNAME
prod-gw,partner-gw   llm.prod.example.com,llm.partner.example.com

IngressReady waits for all of a route's Gateways, and turns True once both prod-gw and partner-gw accept the routes. If partner-gw is missing or no controller manages it, IngressReady stays False with the reason ParentStatusNotAvailable, and so does the InferenceService's Ready condition. To check each Gateway, see A route is missing from a gateway.

Troubleshooting¶

Routes still attach to the cluster's Gateway¶

When the routes in prod still name shared-gw, check for these causes:

The entry's key isn't prod, the InferenceServices' namespace.
The entry has no primary.omeIngressGateway. An entry with only additional keeps the cluster's primary Gateway.
The InferenceService has an ome.io/ingress-gateway annotation, even an empty one. kubectl get inferenceservice llama-chat -n prod -o jsonpath='{.metadata.annotations}' prints its annotations.
You edited the ConfigMap instead of upgrading the release. Restart the controller, which then updates all the InferenceServices: kubectl rollout restart deployment ome-controller-manager -n ome.

IngressReady is False¶

Read the reason and message of the InferenceService's IngressReady condition:

Shell

kubectl get inferenceservice llama-chat -n prod \
  -o 'custom-columns=REASON:.status.conditions[?(@.type=="IngressReady")].reason,MESSAGE:.status.conditions[?(@.type=="IngressReady")].message'

For example, when a Gateway hasn't accepted the engine route yet:

Output

REASON                     MESSAGE
ParentStatusNotAvailable   Engine HttpRouteNotReady

The message starts with the route that isn't ready: Engine for llama-chat-engine, TopLevel for llama-chat. When a Gateway turns a route down, the reason and message come from the Gateway. These are the reasons you're most likely to see:

Reason	Cause	Fix
`ParentStatusNotAvailable`	A Gateway on the route, primary or additional, hasn't accepted it yet. If this lasts, the Gateway is missing or misspelled, or no controller manages it.	Compare the route's Gateways with those that `kubectl get gateway -A` lists.
`NotAllowedByListeners`	No listener on the Gateway allows routes from `prod`.	Allow `prod` in a listener's `allowedRoutes`, as in Before you begin.
`NoMatchingListenerHostname`	No listener's hostname matches the route's host.	Set a listener hostname that matches the host, as in Before you begin, or fix the entry's `ingressDomain`.

When IngressReady is True lists the other reasons.

The controller doesn't start after the upgrade¶

A class that isn't a lowercase RFC 1123 label, or a value of the wrong type, stops the controller from loading its ingress config. A new controller pod logs Failed to initialize ingress configuration and exits. For a bad class, the error starts with invalid ingress config and names the field, such as namespaceIngressGateways["prod"].primary.class. An additional written as a map gives unable to parse ingress config json instead.

Controller pods that are still running hit the same error. They stop creating and updating routes, and serving workloads that run as Deployments or, in the deprecated MultiNode mode, LeaderWorkerSets. Fix the values, and run the upgrade in Step 1 again.

Clean up¶

Remove the annotation from llama-chat:

Shell

kubectl annotate inferenceservice llama-chat -n prod ome.io/ingress-additional-gateways-

Output

inferenceservice.ome.io/llama-chat annotated

To move prod back to the cluster's Gateway, delete its entry from namespaceIngressGateways in your values file, and run the upgrade in Step 1 again. The routes in prod go back to shared-gw, at the host llm.example.com.

Next steps¶

Use multiple gateways to attach the cluster's routes to more than one Gateway.
Configure ingress for the other ingress settings.

Use per-namespace gateways Unreleased: coming in v1.3

Step 1: Map prod to its Gateway¶

Step 2: Check the routes¶

Override the Gateways for one InferenceService¶

Troubleshooting¶

Routes still attach to the cluster's Gateway¶

IngressReady is False¶

The controller doesn't start after the upgrade¶

Clean up¶

Next steps¶

Step 1: Map `prod` to its Gateway¶