Set Instance readiness deadlines Unreleased: coming in v1.3

On OMENative components, each attempt to create, update or restart an Instance has a deadline to become Ready. A bad image, a crashing server or a pod that no node can take then ends in a Failed Instance that says why, instead of a wait with no end. The deadline acts on the whole Instance: one pod, or a leader and its workers.

The ome-resources chart sets all three limits by default. You'll give one component more time, watch its Instance against the deadline, and change the cluster defaults.

How the deadlines work¶

A component's value in the InferenceService wins over the runtime's, and either wins over the cluster's. The same timeout sets the deadline of an Instance migration, and a drain that runs past it raises DrainOverdue. OME also derives a multi-pod Instance's gang schedule timeout from it.

The clock stops while an attempt waits on something outside the Instance: a scheduling gate, a quota refusal, an unschedulable pod, an unreachable node or a paused rollout. When the wait ends, the attempt gets a full timeout again. A changed timeout applies to new attempts, and to running ones that have no deadline yet. A changed grace period applies at once. Operations has the details.

Step 1: Give a component more time¶

Create a namespace for the example:

kubectl create namespace qwen3-deadlines

namespace/qwen3-deadlines created

This InferenceService runs the engine on OMENative and gives it 45 minutes, to leave room for a first image pull on a new node:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-deadlines
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 1
    maxReplicas: 1
    lifecycle:
      instanceReadyTimeout: 45m

Apply it:

kubectl apply -f qwen3-0-6b-deadlines.yaml

inferenceservice.ome.io/qwen3-0-6b created

To set a default for every InferenceService that uses a runtime, put the same lifecycle block under the runtime's engineConfig, decoderConfig or routerConfig.

Once OME has created the engine's InferenceReplica, qwen3-0-6b-engine, check the timeout that applies:

kubectl get inferencereplica qwen3-0-6b-engine -n qwen3-deadlines \
  -o jsonpath='{.status.conditions[?(@.type=="InstanceReadyTimeoutUnconfigured")].message}{"\n"}'

Instance readiness deadline is 45m0s

Without a timeout, the condition is True, and its message starts with no instanceReadyTimeout is set: see The condition says no timeout is set. OME creates an InferenceReplica only for components on OMENative. If kubectl can't find it, see Opt in to OMENative.

Step 2: Watch the Instance against its deadline¶

Right after you apply the InferenceService, OME starts a Create operation for its Instance. Show the Instance in the wide format, which includes the operation's times:

kubectl ome instance status qwen3-0-6b 0 --component engine -n qwen3-deadlines -o wide

These rows show the attempt and its deadline:

Once the Instance is Ready, the operation rows are gone.

When a limit runs out¶

OME marks the Instance Failed, and the failure reason row says why:

What OME does next depends on the cause and the operation:

• Only an image or configuration error counts against the revision. OME retries the revision with backoff, then holds it until you publish a fixed one or release it.
• A failed Create or single-pod update then starts a new attempt. If the revision isn't at fault and the pods all ran on one node, OME can build it on another node, up to 3 times with the chart.
• When a multi-pod update fails under the default SurgeThenDrain strategy, OME removes the replacement, and the Instance goes back to Ready on its old revision. Under RecreatePod or an in-place strategy, the Instance stays Failed until a new revision reaches it, and a reset skips it.
• A failed restart leaves the Instance Failed, with its pods, until you reset it or a new revision reaches it.

What OME does with a failed Instance has the details. OME also records an InstanceFailed Warning event on the InferenceService. List those events:

kubectl get events -n qwen3-deadlines --field-selector reason=InstanceFailed

While no Instance has failed, kubectl prints:

No resources found in qwen3-deadlines namespace.

Each message names the Instance, its pod and the reason, and often what OME did next.

Change the cluster defaults¶

Set the cluster values under ome.controller.lifecycle, in the values file you install ome-resources with:

ome:
  controller:
    lifecycle:
      instanceReadyTimeout: 30m
      stuckPodGracePeriod: 2m
      unschedulableGracePeriod: 15m

Give each duration a unit, as in 90s or 1h. A bare number, like 30, stops new controller pods from starting: see The controller doesn't start after you change the settings. To turn a setting off, set it to null.

With no timeout at all, a migration request fails as soon as OME accepts it, with deadline exceeded in phase Accepted: surge never allocated. This is a known bug, so keep a timeout set while you use migrations.

Keep your other values in the file: helm upgrade resets the values that aren't in the file to the chart's defaults. Upgrade with the chart version that you already run, which helm list -n ome shows in its CHART column:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that the release has been upgraded, and the controller's pods restart with the new settings. Wait for the rollout:

kubectl rollout status deployment ome-controller-manager -n ome

deployment "ome-controller-manager" successfully rolled out

Then read the settings back from the inferenceservice-config ConfigMap:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.lifecycle}' \
  | jq '{instanceReadyTimeout, stuckPodGracePeriod, unschedulableGracePeriod}'

{
  "instanceReadyTimeout": "30m",
  "stuckPodGracePeriod": "2m",
  "unschedulableGracePeriod": "15m"
}

A setting you turned off shows as null. A kustomize install sets only instanceReadyTimeout. To edit the ConfigMap directly instead, see Change ConfigMap settings.

Troubleshooting¶

The condition says no timeout is set¶

The InstanceReadyTimeoutUnconfigured condition is True, though you set a timeout. Check the places OME reads:

• In the InferenceService, the value goes under the component, spec.engine.lifecycle, not under spec.lifecycle.
• A 0s in the InferenceService counts as unset, and hides the runtime's value.
• In the cluster settings, the value needs a unit and must be above zero. OME treats "30", 0s and -5m as unset.

To read the cluster settings, use the ConfigMap check in Change the cluster defaults.

The Instance is stuck and never fails¶

An Instance stays in Creating, or another phase, past its deadline. Check its rows, as in Step 2:

The controller doesn't start after you change the settings¶

After a helm upgrade, the rollout doesn't finish, and the new controller pods crash-loop. Read the logs of every replica:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

A pod that failed to start logs Failed to initialize lifecycle scale configuration, with an error that contains unable to parse lifecycle config json. The lifecycle entry isn't valid JSON, or a duration in it is a bare number. Give each duration a unit, and upgrade again. Until then, the replicas that were already running ignore the whole lifecycle entry, so only a component's own instanceReadyTimeout applies.

Clean up¶

Delete the InferenceService and its namespace:

kubectl delete inferenceservice qwen3-0-6b -n qwen3-deadlines
kubectl delete namespace qwen3-deadlines

inferenceservice.ome.io "qwen3-0-6b" deleted from qwen3-deadlines namespace
namespace "qwen3-deadlines" deleted

If you changed the chart values, put back the ones you want to keep in your values file, and upgrade the release again with the same --version.

Next steps¶

• Reset failed Instances: start over an Instance that failed during a create or a restart.
• Release a held revision: let OME try a revision again after it's held.
• Instances: the phases and operations of an Instance.
• Troubleshoot an InferenceService: find why an InferenceService doesn't serve.