Instance restart policy Unreleased: coming in v1.3

When a pod of a ready Instance fails, its component's lifecycle.restartPolicy decides what OME repairs: only that pod, or the whole Instance. By default, OME rebuilds a multi-pod Instance whole, so that a leader and its workers come back as one group. For a single-pod Instance, it repairs only the failed pod. Only OMENative components read the field.

The two policies¶

The policy is separate from the pod's own restartPolicy, which is Always by default, so the kubelet restarts a crashed container in place under either policy.

The pods of a multi-pod Instance run one copy of the model as a single process group, and a restart of any of them breaks the group. A single-pod Instance has no group to break, and a restart in place is the quicker repair.

Set RecreateInstanceOnPodRestart on a single-pod component when a crashed engine should start over in a new pod, which runs its init containers again and gets fresh emptyDir volumes. Set None on a multi-pod component only if its engine can take a restarted or replaced pod back into a running group.

Set the policy¶

The engine in Serve a model on OMENative has one pod, so its default policy is None. This version of the guide's InferenceService rebuilds the engine's Instance in a new pod when its container restarts after the Instance is ready:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-native
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 1
    maxReplicas: 1
    lifecycle:
      restartPolicy: RecreateInstanceOnPodRestart

If you still run the guide's InferenceService, apply the file:

kubectl apply -f qwen3-0-6b-native.yaml

inferenceservice.ome.io/qwen3-0-6b configured

Changing the policy doesn't start a rollout. OME uses the new value from then on, and a restart already under way still finishes.

OME takes lifecycle.restartPolicy from the first of these that has one:

1. The component in the InferenceService: spec.engine, spec.decoder or spec.router.
2. The component's config in the serving runtime: spec.engineConfig, spec.decoderConfig or spec.routerConfig.
3. The default for the Instance's shape, from the table above.

There's no cluster-wide setting. A runtime sets the policy for the InferenceServices that use it and don't set their own:

spec:
  engineConfig:
    lifecycle:
      restartPolicy: None

LifecycleSpec lists the other lifecycle fields.

Choose a policy per component¶

Each component takes its own policy. This prefill-decode InferenceService serves llama-3-2-1b-instruct with the runtime srt-llama-3-2-1b-instruct-pd, both from OME's catalog. The runtime needs a GPU node with RDMA networking for each engine and decoder pod: see PD mode.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-pd
  namespace: llama-demo
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: srt-llama-3-2-1b-instruct-pd
  engine:
    minReplicas: 2
    maxReplicas: 2
    lifecycle:
      restartPolicy: RecreateInstanceOnPodRestart
      updateStrategy:
        type: SurgeThenDrain
        rollingUpdate:
          maxUnavailable: 1
  decoder:
    minReplicas: 1
    maxReplicas: 1
    lifecycle:
      restartPolicy: None
  router:
    minReplicas: 1
    maxReplicas: 1

• The engine runs prefill. When a pod of a ready prefill Instance fails, OME rebuilds the Instance in a new pod. maxUnavailable: 1 keeps the EngineReady condition True while no more than one Instance is out of service: see Observe a restart.
• The decoder states its default, None, which also overrides any value in the runtime. OME repairs only the decoder pod that fails.
• The router keeps its default, None.

What triggers a restart¶

Under RecreateInstanceOnPodRestart, OME restarts a Ready Instance when:

• The main container of one of its pods, ome-container, restarts. Restarts of sidecars and init containers don't count.
• One of its pods is in phase Failed.
• It has fewer pods than it needs, for example because a pod was deleted or evicted.

So by default, a node drain that evicts one pod of a multi-pod Instance rebuilds the whole Instance: see Configure pod disruption budgets.

Before an Instance is Ready, a crashing container is part of an ordinary start, and the Instance's readiness deadline decides when it fails.

The crash-loop repair¶

Some pod states, such as CrashLoopBackOff and ImagePullBackOff, don't clear on their own. When a pod of a Ready Instance is in one of them and is older than the stuck-pod grace period, OME restarts the whole Instance, under either policy. Its RestartTriggered event says the pod is wedged in <reason> past the stuck-pod grace.

The ome-resources chart sets the grace period, ome.controller.lifecycle.stuckPodGracePeriod, to 60s. Without the setting, the repair is off.

The chart also sets ome.controller.lifecycle.repairBatchSize, to 10: how many of these repairs OME opens per component in one reconcile pass. The bound counts Instances, so a leader and its workers count as one. Only a fresh crash-loop repair counts against it: a repair that's already running, a restart that the policy starts and the replacement of pods that are already gone don't count, and don't wait. Instances past the bound are repaired on the following passes, with no RepairHeld event while they wait. Without the setting, OME opens every repair in one pass. The manager reads the value once, when it starts, and won't start on zero or a negative value, so change it with a Helm upgrade, as Change ConfigMap settings describes.

maxUnavailable holds only a repair that takes a serving Instance offline: at maxUnavailable: 1, such a repair starts only while no other Instance of the component is restarting or out of service for an update. A repair of an Instance whose pods serve nothing starts without waiting, and no repair waits for its rollout group: a wedged Instance is already out of the group's serving count, so holding its repair would only prolong the outage it ends. When maxUnavailable keeps a component from opening any repair in a pass, a RepairHeld warning event names the Instance that waits and the budget. Restarts that the policy starts don't wait for maxUnavailable. Repairs honor it under every update strategy, SurgeThenDrain included.

What a restart does¶

OME sets the Instance's phase to Restarting. It takes the pods out of the Service, deletes them once the Service stops routing to them, and creates them again on the same revision, one incarnation higher. Once the new pods are ready and have passed the component's minReadySeconds, the Instance is Ready again.

The old pods' logs go with them. When a pod failed, the Instance's lastFailure keeps its container, reason, exit code and time, even after the Instance is Ready again.

A restart has until the Instance readiness timeout to get its new pods serving: the component's lifecycle.instanceReadyTimeout, or 30m with the ome-resources chart. When the time runs out, or a new pod stays stuck past the stuck-pod grace period, the Instance becomes Failed. OME keeps a Failed Instance and its pods for you to inspect until you reset it or a new revision reaches it.

Observe a restart¶

OME records a restart's events on the InferenceService:

For example, after the engine's container is killed for running out of memory, list the restarts:

kubectl get events -n qwen3-native --field-selector reason=RestartTriggered \
  -o custom-columns=KIND:.involvedObject.kind,NAME:.involvedObject.name,MESSAGE:.message

KIND               NAME         MESSAGE
InferenceService   qwen3-0-6b   OMENative component=engine instance=0 restart triggered: pod qwen3-0-6b-engine-0-default-0 container ome-container restarted after Ready: OOMKilled (exit 137) (incarnation=2)

kubectl ome instance shows the restart too. In the output of kubectl ome instance list qwen3-0-6b -n qwen3-native, PHASE is Restarting and IDX/INC shows the new incarnation, such as 0/2. The kubectl ome instance status command adds the operation, the last failure and each pod's restarts.

By default, while an engine Instance restarts, the InferenceService's EngineReady condition is False, with the reason InsufficientAvailable. With maxUnavailable set, it stays True, with MinimumAvailable, while no more than that many Instances are out of service: see Conditions.

Updates, migrations and pauses¶

Next steps¶

• Instances: phases, incarnations and lastFailure.
• Set Instance readiness deadlines: decide when a slow or stuck Instance fails.
• Reset failed Instances: start over Instances that OME marked Failed.
• OMENative update strategies: how a change to the pod template replaces an Instance's pods.
• Serve a multi-node model: run a multi-pod engine, which OME rebuilds whole by default.