Recover stuck deletions Unreleased: coming in v1.3

When a node dies, its pods can stay Terminating, and stall any scale-down, update, migration or teardown of an OMENative component that waits for them. Turn on the force-delete escalation, and OME deletes such a pod once the cluster shows that its node is dead. The teardown deadline, 30m in the chart, lets a deleted InferenceService's InferenceReplicas go even when pods remain. If a pod is stuck now, go to Manual recovery.

How OME recovers a stuck pod¶

OMENative gives each pod a stable name, and reuses it when it recreates the pod. So while an old pod stays Terminating, the recreated pod waits.

With the escalation on, OME force-deletes a Terminating pod when all three hold:

1. It's more than overdueSlack past its deletionTimestamp, which already includes its grace period.
2. It has no finalizers.
3. Its node is dead, by one of these checks:

A node whose Ready condition is True never counts as dead, whatever its taints. OME looks for such pods when it scales a component down or tears it down, and when it updates or migrates an Instance. Once the pod is gone, the work that waited for it goes on.

Step 1: Turn on the force-delete escalation¶

These are operator settings for every OMENative component in the cluster, not fields of a component's lifecycle block. The chart writes them from ome.controller.lifecycle into the lifecycle key of the inferenceservice-config ConfigMap in ome:

The escalation is on when both forceDelete durations are set and valid. Add them to your values file:

ome:
  controller:
    lifecycle:
      forceDelete:
        overdueSlack: 2m
        nodeUnreachableThreshold: 5m

Give each duration a unit: a bare number, like 120, stops the new manager pods from starting (see The controller doesn't start after the upgrade). To turn the escalation off, remove the block and upgrade.

If you installed from the config/ manifests, add the block as JSON to the ConfigMap's lifecycle key instead, with kubectl edit configmap inferenceservice-config -n ome. OME picks up the edit without a restart. OME ignores the whole key while it holds invalid JSON, and new manager pods exit until you fix it.

On a chart install, keep your other values in the file, since helm upgrade resets any that are missing. Upgrade with the chart version that you run, from the CHART column of helm list -n ome:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that the release ome has been upgraded, and the manager pods restart. Wait for the rollout:

kubectl rollout status deployment ome-controller-manager -n ome

deployment "ome-controller-manager" successfully rolled out

Check the block that OME reads:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.lifecycle}' | jq .forceDelete

{
  "nodeUnreachableThreshold": "5m",
  "overdueSlack": "2m"
}

Step 2: Check the recovery¶

When OME force-deletes a pod, it records a PodForceDeleted Warning event on the InferenceService, or on the InferenceReplica once the InferenceService is gone:

kubectl get events -n qwen3-native --field-selector reason=PodForceDeleted \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

OMENative component=engine instance=0: force-deleted stuck-Terminating pod qwen3-0-6b-engine-0-default-0 on node gpu-node-3 (evidence=node-unreachable-taint, 2m41s past the pod's own deletion deadline)

Then wait for the InferenceReplica to be Ready:

kubectl wait --for=condition=Ready inferencereplica/qwen3-0-6b-engine -n qwen3-native --timeout=15m

inferencereplica.ome.io/qwen3-0-6b-engine condition met

kubectl get configmap qwen3-0-6b-ome-migration-audit -n qwen3-native \
  -o jsonpath='{.data.history\.json}' | jq '.entries[] | select(.outcome == "force-delete-unreachable")'

{
  "requestUUID": "0b6f4f59-3d0e-4c8a-9a52-6f1d2c7e8a41",
  "component": "engine",
  "sourceInstance": 0,
  "phase": "Completed",
  "reason": "ForceDelete",
  "fromNode": "gpu-node-3",
  "startedAt": "2026-09-28T10:14:03Z",
  "completedAt": "2026-09-28T10:14:03Z",
  "outcome": "force-delete-unreachable"
}

Change the teardown deadline¶

When you delete an OMENative InferenceService, Kubernetes deletes its InferenceReplicas too. Each keeps the finalizer ome.io/ir-teardown until OME has drained and deleted the component's pods and PodGroups.

teardown.deadline caps that wait, counted from the InferenceReplica's delete. At the deadline, OME removes the finalizer, and Kubernetes deletes what's left in the background. A pod on a dead node can then stay Terminating until you force-delete it by hand.

Pick a deadline longer than your pods take to drain, plus their grace period, overdueSlack and nodeUnreachableThreshold, so that the escalation removes pods on dead nodes first. The chart's 30m fits its default grace period for serving pods, 600 seconds, and the values from Step 1. For a long grace period, like terminationGracePeriodSeconds: 1800, raise it:

ome:
  controller:
    lifecycle:
      forceDelete:
        overdueSlack: 2m
        nodeUnreachableThreshold: 5m
      teardown:
        deadline: 1h

To hold the finalizer until the teardown is done, with no deadline, set teardown: null instead. Installs from the config/ manifests set no deadline.

Upgrade the release and wait for the rollout, as in Step 1. Then check the block:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.lifecycle}' | jq .teardown

{
  "deadline": "1h"
}

When a teardown reaches the deadline, OME records a TeardownDeadlineExceeded Warning event on the InferenceReplica, with a count of the pods left:

kubectl get events -n qwen3-native --field-selector reason=TeardownDeadlineExceeded \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

teardown deadline 1h0m0s exceeded (1 owned pod(s) observed; selector: parent="qwen3-0-6b", component="engine"); releasing finalizer ome.io/ir-teardown to background GC; manual escape: kubectl patch inferencereplica qwen3-0-6b-engine -n qwen3-native --type=merge -p '{"metadata":{"finalizers":null}}'

Manual recovery¶

Force-delete a stuck pod yourself when the escalation is off or doesn't act on it, or once the teardown deadline has passed. First make sure that the node is down for good: force-deleting a pod on a node that's only cut off leaves its containers running.

kubectl delete pod qwen3-0-6b-engine-0-default-0 -n qwen3-native --grace-period=0 --force

Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "qwen3-0-6b-engine-0-default-0" force deleted from qwen3-native namespace

A pod with finalizers stays until they're removed. Once the pod is gone, the work that waited for it goes on.

If a deleted InferenceReplica still stays, release its finalizer. OME stops cleaning up the component, and Kubernetes deletes what's left in the background:

kubectl patch inferencereplica qwen3-0-6b-engine -n qwen3-native --type=merge -p '{"metadata":{"finalizers":null}}'

inferencereplica.ome.io/qwen3-0-6b-engine patched

Patch only an InferenceReplica that's being deleted: on any other, OME adds the finalizer back.

Troubleshooting¶

The controller doesn't start after the upgrade¶

kubectl rollout status doesn't finish, and the new manager pods exit with Failed to initialize lifecycle scale configuration in their logs. A duration in the lifecycle values is a bare number, like 120, or, on a config/ install, the key holds invalid JSON. Fix the value as in Step 1. Until then, the manager pods that were already running ignore the whole lifecycle key, the escalation and the teardown deadline included.

A pod stays Terminating with the escalation on¶

List the component's pods with their nodes, deletion times and finalizers:

kubectl get pods -n qwen3-native -l ome.io/inferenceservice=qwen3-0-6b,component=engine \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,DELETION:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers

NAME                            NODE         DELETION               FINALIZERS
qwen3-0-6b-engine-0-default-0   gpu-node-3   2026-09-28T10:11:22Z   <none>

Read the node's Ready condition and when it last changed:

kubectl get node gpu-node-3 \
  -o jsonpath='{range .status.conditions[?(@.type=="Ready")]}{.status} since {.lastTransitionTime}{"\n"}{end}'

Unknown since 2026-09-28T10:02:47Z

Then find the cause:

A PodDeleteBlockedByFinalizer event¶

The pod is more than overdueSlack past its deletionTimestamp, but it has finalizers, which OME leaves to the controllers that own them. The event names them:

kubectl get events -n qwen3-native --field-selector reason=PodDeleteBlockedByFinalizer \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

OMENative component=engine instance=0: pod qwen3-0-6b-engine-0-default-0 is 3m5s past its own deletion deadline but pinned by finalizers [example.com/cleanup]; OME never strips another controller's finalizer — the finalizer owner must resolve it

Fix the controller that owns the finalizer. Remove the finalizer yourself only if its controller is gone for good, since its cleanup then never runs.

An Instance waits with NodeUnknown¶

A pod that holds a name the Instance needs is on a node whose kubelet stopped reporting it, and its containers may still run. Such a pod has one of two shapes:

• It's in phase Unknown.
• It's Running with its Ready condition False, while ContainersReady is True and every readiness gate it declares is satisfied. When a node stops heartbeating, the node lifecycle controller withdraws Ready from every pod on it, so a dead node usually leaves this shape.

With the escalation on, OME force-deletes a pod of either shape once its node counts as dead, by the same node checks. overdueSlack doesn't apply, since no one has asked to delete the pod. List the PodForceDeleted messages as in Step 2:

OMENative component=engine instance=0: force-deleted pod qwen3-0-6b-engine-0-default-0 on node gpu-node-3, which its kubelet stopped reporting (evidence=node-unreachable-taint)

A pod that's also Terminating follows the usual rules instead, overdueSlack included. A Running pod that isn't Ready on a node whose Ready condition is True doesn't hold the Instance: that's its kubelet catching up on a readiness write, and OME leaves the pod alone.

Without the escalation, only a pod in phase Unknown raises NodeUnknown, and the Instance waits until the node comes back, or until you force-delete the pod by hand. A Running pod that lost Ready doesn't raise it: telling a dead node from a kubelet's lag takes the node read that comes with the escalation, so the Instance just isn't Ready.

A deleting InferenceReplica has TeardownBlocked events¶

The teardown has no valid deadline, and pods or PodGroups remain. OME records this event while it waits, so it also shows during an ordinary teardown. List the messages:

kubectl get events -n qwen3-native --field-selector reason=TeardownBlocked \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

teardown blocked: 1 owned pod(s) and 0 owned PodGroup(s) remain (selector: parent="qwen3-0-6b", component="engine"); finalizer ome.io/ir-teardown holds until cleanup completes (no lifecycle.teardown.deadline configured); manual escape: kubectl patch inferencereplica qwen3-0-6b-engine -n qwen3-native --type=merge -p '{"metadata":{"finalizers":null}}'

If the part in parentheses says lifecycle.teardown.deadline configured but invalid, fix the deadline as in Change the teardown deadline. Otherwise, check the pods that remain as in A pod stays Terminating with the escalation on.

DrainOverdue on the InferenceReplica¶

An Instance that OME is removing is still draining past its deadline: the start of the delete, plus the component's instanceReadyTimeout. DrainOverdue is only a warning, and the drain goes on. The event names the pods that are still Terminating or draining:

kubectl get events -n qwen3-native --field-selector reason=DrainOverdue \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

OMENative component=engine instance=0: drain overdue by 4m12s (deadline 2026-09-28T10:44:03Z); 1 pod(s) still Terminating (qwen3-0-6b-engine-0-default-0)

For a pod that's still Terminating, check it as in A pod stays Terminating with the escalation on.

Next steps¶

• Set Instance readiness deadlines, which also set the drain deadlines behind DrainOverdue.
• Reset failed Instances.
• Request an Instance migration, to move an Instance off a node before you take the node down.
• Troubleshoot an InferenceService.