Publish a global endpoint Preview Unreleased: coming in v1.3

Give an alpha multi-cluster InferenceService one global hostname that splits its traffic across workload clusters by its TrafficMap's weights.

A global endpoint gives a multi-cluster InferenceService one hostname for its clients, whichever workload clusters serve it. OME writes the weights of the service's TrafficMap into a Gateway API HTTPRoute on the control-plane cluster, and your Gateway there splits requests across the workload clusters by those weights. The examples publish the InferenceService chat, in the namespace prod, at chat.prod.global.example.com.

Multi-cluster serving, routing and publishing are alpha, still in development, and off by default. Their APIs and settings can change without notice.

Before you begin¶

A control-plane cluster running OME v1.3 or later, installed with the ome-resources chart as the release ome in the namespace ome. The release's values set ome.multicluster.enabled=true and ome.multicluster.role=control-plane, as in Controller manager flags, and ome.multicluster.config.routing.enabled=true, as in Turn on routing.
Helm, and kubectl with the control-plane cluster as its current context.
The Gateway API CRDs there, and an implementation that accepts ExternalName Services as backends and a URLRewrite filter on a single backend. The Gateway API leaves both to each implementation, and advises against ExternalName backends, so check yours.
A Gateway for the global hostnames, here global in gateway-system, with an HTTP listener on port 80 whose hostname is *.global.example.com or unset. DNS records, such as a wildcard, point the hostnames at it, and the listener's allowedRoutes admits routes from prod: by default, a listener admits only its Gateway's namespace.
Two registered WorkloadClusters, here worker-a and worker-b.
On each, OME's ingress creation turned on, with Gateway API routes, per-service subdomains and a domain of its own, as in Switch schemes cluster-wide. The control-plane Gateway has to reach chat there on port 80, at http://chat.prod.worker-a.example.com/ and http://chat.prod.worker-b.example.com/. If it can't resolve those hostnames, see EndpointSlice fallback.
The InferenceService chat in prod, with no router or decoder, and eligible for routing. Its spec.placement.mode is All or Split, because the default, Single, places it on one workload cluster. Once it's placed on both workload clusters, kubectl get trafficmap chat -n prod -o jsonpath='{.spec.entries[*].cluster}' prints worker-a worker-b.

How publishing works¶

For each TrafficMap, the controller's built-in publisher, gatewayapi, writes these objects on the control-plane cluster, by default in the InferenceService's namespace, with labels such as ome.io/placement-endpoint-isvc: chat:

An HTTPRoute named after the InferenceService, here chat-global, attached to the globalGateway Gateway. It matches the global hostname at every path, and has a backend for each TrafficMap entry, at the entry's weight, 0 included.
For each backend, an ExternalName Service named tmb- and a hash. It points at the host of the URL that the entry's workload cluster reports for the service, such as chat.prod.worker-a.example.com.

The route keeps the client's path, so http://chat.prod.global.example.com/v1/models reaches the engine as sent. A service with a router or a decoder belongs on the shared host instead, as Per-service subdomains warns, and its clients add its path prefix, such as /prod/chat/.

The map's Published condition means that the publisher wrote the route. The route's own status says whether the Gateway accepted it, and Step 2 checks both.

Step 1: Configure publishing¶

Set the publisher's settings under ome.multicluster.config.endpoint, in the values file that you install ome-resources with:

values.yaml

ome:
  multicluster:
    enabled: true
    role: control-plane
    config:
      endpoint:
        globalGateway: gateway-system/global
        backendPort: 80
        globalHostTemplate: "{{ .Name }}.{{ .Namespace }}.global.example.com"
        gatewayBackend:
          rewriteHostname: true
      routing:
        enabled: true

Keep the other values that you installed with, such as a probe from Configure routing health probes, in the same file. Otherwise helm upgrade sets them back to the chart's defaults.

Value	Default	What it sets
`globalGateway`	Empty	The Gateway for the routes, as `namespace/name`, or `name` in the route's namespace.
`backendPort`	`0`	The workload clusters' Gateway port that the routes send to. Required with `globalGateway`.
`globalHostTemplate`	Empty	The global hostnames. See Choose the global host.
`gatewayBackend.rewriteHostname`	`false`	`true` gives each backend a `URLRewrite` filter that sets the `Host` header to its workload hostname.

This setup needs rewriteHostname: OME's routes on a workload cluster match only that cluster's hostnames, so a request with the Host header chat.prod.global.example.com gets HTTP 404 there. Optional settings covers the other endpoint values.

Choose the global host¶

globalHostTemplate is a Go template that can use only the InferenceService's {{ .Name }} and {{ .Namespace }}. Quote it in YAML, and keep double quotes and backslashes out of it: the chart copies it into JSON unescaped.

The annotation ome.io/global-host on an InferenceService wins over the template, and takes effect without a restart. With the chart's default, an empty template, only annotated InferenceServices get a hostname:

YAML

metadata:
  annotations:
    ome.io/global-host: chat.global.example.com

Either way, the hostname has to:

Be a valid, lowercase DNS subdomain name, which can start with *., and not an IP address, or Published gets the reason InvalidPlan.
Match a listener of the Gateway, here *.global.example.com, and resolve to the Gateway in DNS.
Be free: if another route on the Gateway lists exactly the same hostname, the reason is ClaimRejected.

Upgrade the release¶

Upgrade the release with the chart version you already run, 1.3.0 here:

Shell

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

The chart restarts the controller, which reads the settings only when it starts. Wait for the rollout:

Shell

kubectl rollout status deployment ome-controller-manager -n ome

Output

deployment "ome-controller-manager" successfully rolled out

If it fails, see The controller rollout doesn't finish. Then check the route's hostname and Gateway, and each backend's Host header and weight:

Shell

kubectl get httproute chat-global -n prod -o jsonpath='{.spec.hostnames[0]}{" "}{.spec.parentRefs[0].namespace}{"/"}{.spec.parentRefs[0].name}{"\n"}{range .spec.rules[0].backendRefs[*]}{.filters[0].urlRewrite.hostname}{" weight="}{.weight}{"\n"}{end}'

Output

chat.prod.global.example.com gateway-system/global
chat.prod.worker-a.example.com weight=1
chat.prod.worker-b.example.com weight=1

Step 2: Check the publication¶

Check the map's Published condition:

Shell

kubectl get trafficmap chat -n prod -o jsonpath='{range .status.conditions[?(@.type=="Published")]}{.status}{" "}{.reason}{": "}{.message}{"\n"}{end}'

Output

True Published: TrafficMap is published by publisher "gatewayapi"

If it isn't True, see Troubleshooting. Then check that the Gateway accepted the route:

Shell

kubectl get httproute chat-global -n prod -o jsonpath='{range .status.parents[*].conditions[*]}{.type}{"="}{.status}{" "}{.reason}{"\n"}{end}'

The output should include Accepted=True and ResolvedRefs=True. If it doesn't, see Requests to the global hostname fail. Last, send a request to the global hostname:

Shell

curl -s http://chat.prod.global.example.com/v1/models

The engine on one of the workload clusters answers with its list of models.

Optional settings¶

After changing one of these settings, rerun the helm upgrade command, and wait for the rollout.

Backend TLS¶

When the workload clusters' Gateways accept only HTTPS, turn on tls. The publisher then writes a BackendTLSPolicy for each backend Service, so that the control-plane Gateway uses TLS and checks the workload hostname's certificate against the system's trusted CAs. It needs:

The BackendTLSPolicy CRD at gateway.networking.k8s.io/v1, in the standard channel from Gateway API v1.4.0, and an implementation that supports it. Without the CRD, the reason is ClaimRejected.
A certificate on each workload cluster's Gateway for its workload hostnames, from a CA that the control-plane Gateway's implementation trusts as a system CA.
backendPort set to the HTTPS port, such as 443.

YAML

gatewayBackend:
  rewriteHostname: true
  tls:
    enabled: true
    wellKnownCACertificates: System

Keep rewriteHostname on: the policy sets the TLS server name, not the Host header. Turning tls off deletes the policies. Check them after the upgrade:

Shell

kubectl get backendtlspolicies -n prod -l ome.io/placement-endpoint-isvc=chat -o jsonpath='{range .items[*]}{.metadata.labels.ome\.io/placement-cluster}{" "}{.spec.validation.hostname}{"\n"}{end}'

Output

worker-b chat.prod.worker-b.example.com
worker-a chat.prod.worker-a.example.com

EndpointSlice fallback¶

When the control-plane Gateway can't resolve the workload hostnames, turn on endpointSlices. The publisher then copies the IP addresses from the status.addresses of each workload cluster's serving Gateway into an EndpointSlice for the backend Service. The Services stay ExternalName, so this helps only an implementation that accepts them and reads their EndpointSlices. It needs:

IP addresses in those status.addresses. Hostname addresses don't count.
Permission for each WorkloadCluster's credentials to get gateways and httproutes in gateway.networking.k8s.io. On a workload cluster, the ome-resources value ome.multiclusterAccess.enabled: true grants it through the ClusterRole ome-multicluster-access to the identities in ome.multiclusterAccess.subjects.

YAML

gatewayBackend:
  rewriteHostname: true
  endpointSlices:
    enabled: true
    addressRefreshInterval: 1m

addressRefreshInterval is required: the publisher rereads the addresses at least that often.

Check the EndpointSlices after the upgrade. The example Gateways report 203.0.113.10 on worker-a and 198.51.100.10 on worker-b:

Shell

kubectl get endpointslices -n prod -l ome.io/placement-endpoint-isvc=chat -o jsonpath='{range .items[*]}{.metadata.labels.ome\.io/placement-cluster}{" "}{.endpoints[*].addresses[0]}{"\n"}{end}'

Output

worker-b 198.51.100.10
worker-a 203.0.113.10

Route namespace¶

routeNamespace puts all the routes and their objects in one namespace, which the Gateway's listener has to admit. The route of an InferenceService from another namespace gets a hash in its name: chat in prod gets chat-ba4cb5d6-global.

Troubleshooting¶

For InvalidPlan, ClaimRejected and ApplyFailed, the controller's log has the error:

Shell

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'TrafficMap publisher reconciliation failed'

The publisher retries on its own: a failed read or write with backoff, and other failures at its resync interval, routing.publisher.resyncInterval (1m in the chart). It also retries when the map or the InferenceService changes. After a fix, check Published again.

The controller rollout doesn't finish¶

The controller exits at startup when a multi-cluster setting is unusable, so kubectl rollout status fails at the deployment's progress deadline, 10 minutes by default. The log names each unusable setting:

Shell

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'invalid multi-cluster configuration'

Error starts with	Fix
`endpoint.backendPort:`	Set `backendPort`, from 1 to 65535, whenever `globalGateway` is set.
`endpoint.globalGateway:`	Set `globalGateway`, or turn the `gatewayBackend` settings off.
`endpoint.gatewayBackend.tls.`	Set both `tls.enabled: true` and `wellKnownCACertificates: System`, or neither.
`endpoint.gatewayBackend.endpointSlices.`	Set both `endpointSlices.enabled: true` and an `addressRefreshInterval` with a unit, such as `1m`, or neither.

Published is False with reason Withdrawn¶

The publisher withdraws a map, and deletes its route and Services, when globalGateway or backendPort isn't set, or the InferenceService has no global hostname. Without the annotation, the template has to render a hostname.

The controller reads the values from the multicluster entry of the inferenceservice-config ConfigMap when it starts. Check them there:

Shell

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.multicluster}' | grep -E '"(globalGateway|backendPort|globalHostTemplate)"'

Output

    "globalHostTemplate": "{{ .Name }}.{{ .Namespace }}.global.example.com",
    "globalGateway": "gateway-system/global",
    "backendPort": 80,

Published is False with reason InvalidPlan¶

Error contains	Likely cause	Fix
`global hostname:`	The hostname isn't a valid DNS subdomain name, or is an IP address.	Fix the annotation or the template.
`parse globalHostTemplate` or `render globalHostTemplate`	The template doesn't parse, or uses a field other than `.Name` and `.Namespace`.	Fix the template.
`Gateway API global gateway`	`globalGateway` isn't `name` or `namespace/name`.	Fix it.
`at most 16 backendRefs`	The map has more than 16 entries, but a route rule holds at most 16 backends.	Place the service on 16 workload clusters or fewer.
`still has legacy endpoint finalizer`	The older publisher hasn't cleaned up this InferenceService.	See LegacyLifecycleActive.

Published is False with reason ClaimRejected¶

Error contains	Likely cause	Fix
`belongs to another source`	An object that the publisher didn't write for this InferenceService has one of its names.	Delete or rename it.
`is already published by HTTPRoute`	Another route on the Gateway has the same hostname, perhaps the publisher's old one after a `routeNamespace` change.	Change one hostname, or delete the old route.
Starts with `read` or `list`	The publisher can't read an object that it checks, for example a BackendTLSPolicy when its CRD is missing.	Install the CRD, or fix the API server error.

Published is False with reason ApplyFailed¶

Writing or deleting one of the publisher's objects failed, and the log has the API server's error. With the EndpointSlice fallback, the error can start with resolve backend addresses for cluster "worker-b" and contain one of these:

Error contains	Likely cause	Fix
`workload cluster is not connected`	The control plane has no working connection to that WorkloadCluster.	Check its `Ready` condition.
`read serving HTTPRoute` or `read serving Gateway`	The route or its Gateway isn't on the workload cluster yet, or the WorkloadCluster's credentials lack `get` permission for it.	Wait for the route, or grant `ome-multicluster-access`.
`has no IP status addresses`	The Gateway reports no IP address.	Wait for one, or turn the fallback off.

Published is False with reason LegacyLifecycleActive¶

The older endpoint publisher, which runs while routing is off and reads the same endpoint values, puts the finalizer ome.io/placement-endpoint on each InferenceService that it publishes. While any InferenceService carries it, the TrafficMap publisher publishes only the maps that it published before. Only the older publisher removes it, so turn routing off until it has. The global hostnames have no route from the first upgrade to the last.

In values.yaml, remove globalGateway and the gatewayBackend settings, and set routing.enabled: false. Keep routeNamespace if you set it: the older publisher looks for its routes there. Upgrade the release. OME deletes the maps, and the older publisher withdraws its routes and removes its finalizers.

Repeat this command until it prints nothing:

Shell

kubectl get inferenceservices -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{" "}{.metadata.finalizers}{"\n"}{end}' | grep 'ome.io/placement-endpoint'

Restore globalGateway, your gatewayBackend settings and routing.enabled: true, and upgrade the release again.

Requests to the global hostname fail¶

Symptom	Likely cause	Fix
The hostname doesn't resolve.	No DNS record.	Add one.
`Accepted` is `False`, reason `NotAllowedByListeners`.	The listener's `allowedRoutes` doesn't admit the route's namespace.	Admit it.
`Accepted` is `False`, reason `NoMatchingListenerHostname`.	No listener's hostname matches the global hostname.	Add a matching listener, or change the hostname.
HTTP 404.	The request reached a Gateway without a route for its hostname and path.	Turn on `rewriteHostname`, as in Step 1. On the shared host, add the path prefix.
HTTP 500, and `ResolvedRefs` is `False`.	The implementation rejects the backends, for example because it doesn't support `ExternalName` Services.	Check its support for `ExternalName` backends.
The connection to a workload cluster fails.	A blocked network path to its Gateway, a wrong `backendPort`, or HTTPS with `tls` off.	Check the path, the port and Backend TLS.

Clean up¶

To stop publishing, remove globalGateway and the gatewayBackend settings from values.yaml, and upgrade the release. The publisher deletes each route and its Services, and routing keeps writing the maps. Keep routeNamespace if you set it, or the old routes stay behind at weight 0 until their maps are deleted.

To stop publishing one InferenceService instead, set spec.routing.enabled: false on it. OME deletes its TrafficMap once the publisher has deleted the route and its Services.

Either way, check that the route is gone:

Shell

kubectl get httproute chat-global -n prod

Output

Error from server (NotFound): httproutes.gateway.networking.k8s.io "chat-global" not found

Next steps¶

Traffic map: how routing works out each cluster's weight, and every reason of the Published condition.
Configure routing health probes: take traffic away from a workload cluster whose endpoint stops answering.
Tune routing capacity polling: lower a workload cluster's weight to the capacity that its endpoints report.
Drain a workload cluster: hold a workload cluster at zero weight for maintenance.
Choose a Gateway API host scheme: the hostnames that each workload cluster gives its InferenceServices.