Configure route timeouts Unreleased: coming in v1.3

Set how long the gateway gives an InferenceService to answer, so long generations and streamed responses can finish. OME puts this request timeout on the Gateway API HTTPRoutes that it creates. You set it for one InferenceService with a component's timeoutSeconds, or for the whole cluster with defaultRouteTimeoutSeconds. The timeout covers the whole request and response, so a stream must finish within it. The chart's default, 0, turns the timeout off.

Where a route's timeout comes from¶

Each HTTPRoute takes its timeout from the component that it sends traffic to, using the first of these that's set:

1. timeoutSeconds on the component: spec.engine, spec.router or spec.decoder.
2. defaultRouteTimeoutSeconds, the cluster-wide default in the ingress entry of the inferenceservice-config ConfigMap.
3. Otherwise, the route has no timeout of its own, and the gateway's default applies: 15 seconds on Envoy Gateway.

Both settings take whole seconds from 0 to 99999. The routes of qwen3-0-6b take their timeouts from these fields:

0 turns the timeout off, while leaving both settings unset gives the gateway's default. The ome-resources chart sets defaultRouteTimeoutSeconds to 0, so after a Helm install, the timeout is off unless a component sets its own. The ConfigMap in the kustomize manifests leaves the setting out, so the gateway's default applies.

Turn the timeout off for long generations and streamed responses whose length you can't predict, or keep a limit longer than your longest request. With 0, OME writes 0s on the route, which Gateway API asks implementations to treat as no timeout, or as the longest one they support.

The ome.io/timeout-idle, ome.io/timeout-max-connection-duration and ome.io/timeout-tcp-connect annotations set connection timeouts, not the request timeout. See Timeout annotations.

Referenced replicas¶

An InferenceService that fronts standalone replicas with spec.replicaRefs has no per-service timeout. The webhook rejects spec.engine, spec.router and spec.decoder next to spec.replicaRefs, with spec.engine and spec.replicaRefs cannot both be set; a service references all of its roles or none, and the referenced InferenceReplicas carry no timeoutSeconds. So every route of such a service takes defaultRouteTimeoutSeconds, or the gateway's default when that's unset.

To allow longer requests, set the cluster-wide default, as in Step 2. After a Helm install, there's often nothing to do: the chart's 0 already turns the timeout off. After a kustomize install, the setting is unset, so Envoy cuts requests at 15 seconds until you add it. The default applies to every InferenceService, so give a declared component that needs a different limit its own timeoutSeconds.

Step 1: Set the timeout for one InferenceService¶

Set timeoutSeconds on a component to give its routes their own timeout. A service with spec.replicaRefs has no components to set: see Referenced replicas. This is the InferenceService qwen3-0-6b from Before you begin, with a 30-minute timeout on the engine:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-0-6b
spec:
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 1
    maxReplicas: 1
    timeoutSeconds: 1800

kubectl apply -f isvc.yaml

inferenceservice.ome.io/qwen3-0-6b configured

This InferenceService has no router, so the engine's value applies to both its routes, qwen3-0-6b-engine and qwen3-0-6b. With a router, also set spec.router.timeoutSeconds: the top-level route sends traffic to the router. Changing timeoutSeconds updates only the routes, so the engine's pods keep running.

List the InferenceService's HTTPRoutes with the timeout of each:

kubectl get httproute -n qwen3-0-6b \
  -o 'custom-columns=NAME:.metadata.name,TIMEOUT:.spec.rules[*].timeouts.request'

NAME                TIMEOUT
qwen3-0-6b          1800s
qwen3-0-6b-engine   1800s

0s means that the timeout is off. <none> means that the route has no timeout of its own, so the gateway's default applies.

To go back to the cluster-wide default, remove timeoutSeconds and apply the file again.

Step 2: Set a cluster-wide default¶

For a cluster-wide limit, set defaultRouteTimeoutSeconds under ome.controller.ingressGateway in the values file that you install ome-resources with. This example allows 10 minutes:

ome:
  controller:
    ingressGateway:
      disableIngressCreation: false
      enableGatewayAPI: true
      omeIngressGateway: envoy-gateway-system/ome-gateway
      defaultRouteTimeoutSeconds: 600

Replace envoy-gateway-system/ome-gateway with your Gateway. Keep your other install values in the file too: helm upgrade resets any value you leave out to the chart's default.

helm list -n ome shows the installed version in its CHART column. Upgrade the release with the chart version you already run, 1.3.0 in this example:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Until v1.3 is released, upgrade from a checkout of main instead.

Helm prints Release "ome" has been upgraded. Happy Helming!. The upgrade restarts the controller, which then applies the new default to every route whose component sets no timeoutSeconds.

With a kustomize install, add "defaultRouteTimeoutSeconds": 600 to the ingress entry of the inferenceservice-config ConfigMap instead, and restart the controller, as Kustomize installs shows. Keep the entry's ingressGateway and ingressService keys, even with Gateway API, or the restarted controller exits.

Check the ConfigMap:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.ingress}' | grep defaultRouteTimeoutSeconds

    "defaultRouteTimeoutSeconds": 600,

Troubleshooting¶

Long requests time out¶

Check the routes with the command in Step 1, and find the TIMEOUT value:

The route keeps its old timeout¶

A component's timeoutSeconds wins over defaultRouteTimeoutSeconds. kubectl get inferenceservice qwen3-0-6b -n qwen3-0-6b -o jsonpath='{.spec.engine.timeoutSeconds}' prints the engine's value, or nothing when it isn't set.

If you edited the ConfigMap by hand, restart the controller to apply the change to every InferenceService:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

If a timeoutSeconds is negative, or either setting is above 99999, the API server rejects the route. The InferenceService can still show Ready, with no event, but the controller's log shows the error:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'fails to reconcile ingress'

The error names the route, as in failed to perform dry-run update for engine HttpRoute qwen3-0-6b-engine, followed by the API server's reason. Set a value in range, and OME retries on its own. To turn the timeout off, use 0.

No HTTPRoutes for the InferenceService¶

If kubectl get ingress -n qwen3-0-6b lists qwen3-0-6b, either enableGatewayAPI is false, or that Ingress is left from before you set it to true and serves until you delete it. For other causes, see OME doesn't create a route.

Next steps¶

• Configure ingress: choose between Kubernetes Ingress and Gateway API, and set the rest of the ingress entry.
• Traffic annotations: the ome.io annotations that tune circuit breaking, retries and connection timeouts for an InferenceService's backends.