Troubleshoot an InferenceService Unreleased: coming in v1.3

When an InferenceService fails to become ready, serve requests or pick up a change, check its status, model, runtime and logs. Then look up what you see in Troubleshooting.

The examples use the InferenceService qwen2-5-7b in the namespace demo. It serves the ClusterBaseModel qwen2-5-7b with a runtime that OME selected, and its engine runs on OMENative as the InferenceReplica qwen2-5-7b-engine and its Instances. OME runs in the namespace ome.

Symptoms and where to start¶

Unless kubectl rejected the InferenceService, start with Step 1, whose report often points at the cause. Then find what you see:

Step 1: Get the overall status¶

kubectl ome status reports on the InferenceService, its pods and their Warning events. Just after you create the InferenceService, the report looks like this:

kubectl ome status qwen2-5-7b -n demo

FIELD                VALUE
Name                 qwen2-5-7b
Namespace            demo
Ready                NotRecorded / Unavailable
Ready reason
Declared runtime
Model                qwen2-5-7b
Generation           1 observed=0; advisory Unverifiable
Pod observation      Reported count=0 truncated=false
Event observation    Reported count=0 truncated=false
engine               NotRecorded / Unavailable; Ready pods=0/0 restarts=0
Rollout              Unknown reported=Unknown
Rollout evidence     Reported / Unverifiable
Autoscaling          Unavailable / Unavailable parent status
Placement            NotConfigured / NotApplicable / NotRecorded
Traffic              Unavailable / Unavailable parent status
Runtime active       Unavailable / Unavailable AutoSelectionNotProbed
Accelerator          Unavailable / Unavailable AutoSelectionNotProbed
Full safe values     Use -o json or -o yaml
Rollout detail       kubectl ome rollout status NAME
Autoscale detail     kubectl ome autoscale status NAME
Placement detail     kubectl ome placement status NAME
Traffic detail       kubectl ome traffic status NAME
Runtime detail       kubectl ome runtime effective NAME
Accelerator detail   kubectl ome accelerator explain NAME

In a healthy report, Ready and each component row start with True / Valid, all pods are ready, and no Issue row appears. Otherwise, read these rows:

Add -o wide for the condition and event messages. The report leaves out Normal events and events on InferenceReplicas: list those with kubectl get events. kubectl ome status describes every row.

Step 2: Check the model¶

Steps 2 and 3 apply when the InferenceService names a base model in spec.model. An InferenceService without one names its runtime in spec.runtime, and OME skips the model fetch and runtime selection: go to Step 4.

Unless the model's weights come from a PVC, OME schedules the pods only onto nodes that hold the weights. Check the model's state and which nodes have it:

kubectl get clusterbasemodel qwen2-5-7b \
  -o 'custom-columns=STATE:.status.state,READY:.status.nodesReady[*],FAILED:.status.nodesFailed[*]'

STATE   READY                   FAILED
Ready   gpu-node-1,gpu-node-2   <none>

For a BaseModel, run kubectl get basemodel qwen2-5-7b -n demo with the same columns. Then act on STATE:

• Ready: at least one node has the model. Other nodes may still be downloading it, so some pods can stay Pending.
• In_Transit: no node has the model yet. The model agent downloads it, and the ome-resources chart installs the agent only when you set modelAgent.enabled: true. See Run the model agent.
• Failed: no node has the model. FAILED lists the nodes whose download failed. See Model lifecycle.

A pvc:// model needs no agent: a metadata Job sets its state. See Serve models from a PVC.

Step 3: Check the runtime¶

kubectl ome runtime explain runs OME's runtime selection for the model and lists every runtime it considered:

kubectl ome runtime explain --isvc qwen2-5-7b -n demo

The first row with Yes in COMPATIBLE is the runtime that OME selects, and each No row gives its reason in REASON. When every row says No, OME leaves the serving workload as it is: read the reasons as Troubleshoot runtime selection explains.

The command finds the model whatever spec.model.kind says, so a Yes row doesn't prove that the kind is right. See An InferenceService that serves a BaseModel gets no pods.

Step 4: Read the logs¶

kubectl ome logs reads the main container, ome-container, of the InferenceService's pods. Read the engine's logs from the last 10 minutes:

kubectl ome logs qwen2-5-7b -n demo --component engine --since 10m

If the command fails with no pods found for InferenceService "qwen2-5-7b" in namespace "demo", check the name and the namespace, then see The InferenceService gets no pods.

For a restarted pod's previous run, use kubectl. List the engine's pods:

kubectl get pods -n demo -l ome.io/inferenceservice=qwen2-5-7b,component=engine

On OMENative, a pod's name includes its Instance, as in qwen2-5-7b-engine-0-default-0 for Instance 0. A Deployment's pods have names like qwen2-5-7b-engine-6c9f8d7b5-x2kqp. Read the previous run of ome-container:

kubectl logs qwen2-5-7b-engine-0-default-0 -n demo -c ome-container --previous

The log usually ends with the error that stopped the container. When a pod of a multi-pod Instance fails, OME recreates all the Instance's pods by default, and their logs are gone: see Instance restart policy. OME still records the Instance's last failure: kubectl ome instance status qwen2-5-7b 0 --component engine -n demo -o wide shows its pod, container, reason and exit code.

Some failures show only in the controller's logs. Read every replica, because only the leader acts, and pass --tail=-1, because a label selector limits kubectl to the last 10 lines per pod:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep qwen2-5-7b

Troubleshooting¶

The webhook rejects the InferenceService¶

kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request: and a message. The webhook checks every create and update, so it can reject an InferenceService that it admitted before, even for a label change. Find the first row that matches:

The warning Runtime srt-qwen2-5-7b will be auto-selected for model qwen2-5-7b is expected. A warning that includes does not declare support for model means that the runtime you named doesn't list the model's format: see kubectl prints a does not declare support warning.

The InferenceService gets no pods¶

kubectl ome status shows Pod observation as Reported count=0 truncated=false. Look for these causes in order, and fix the first that applies. The first three apply when the InferenceService names a base model.

1. spec.model.kind differs from the model's kind. See the next section.
2. A ModelReconcileError Warning event, such as specified base model qwen2-5-7b is disabled. Fix the model that it names.
3. A Normal ModelNotReady event, which the status report leaves out: the model is Sharded, which can't be served in this release. List these events with kubectl get events -n demo --field-selector reason=ModelNotReady. See Distribution.
4. A RuntimeNotFound Warning event: no runtime supports the model. See Troubleshoot runtime selection. An InferenceService that was already serving keeps its pods.
5. An AcceleratorClassError Warning event, or the BestFit accelerator policy. See The InferenceService has an AcceleratorClassError event and BestFit doesn't create or update the workload.
6. The InferenceService lacks an engine section, so OME creates no engine. Add one: a section that sets only minReplicas is enough.

Otherwise, read the controller's logs, as in Step 4.

An InferenceService that serves a BaseModel gets no pods¶

spec.model.kind defaults to ClusterBaseModel. Suppose qwen2-5-7b is a BaseModel in demo, and the InferenceService leaves out the kind. The webhook admits it, but OME leaves the serving workload as it is and records no event or condition. The status report shows Ready as NotRecorded / Unavailable, and the controller's logs show Failed to get referenced ClusterBaseModel. The reverse, kind: BaseModel for a ClusterBaseModel, logs Failed to get referenced BaseModel. Check the kind:

kubectl get inferenceservice qwen2-5-7b -n demo -o custom-columns=KIND:.spec.model.kind,MODEL:.spec.model.name

KIND               MODEL
ClusterBaseModel   qwen2-5-7b

Set the kind to the model's kind, here BaseModel:

kubectl patch inferenceservice qwen2-5-7b -n demo --type=merge -p '{"spec":{"model":{"kind":"BaseModel"}}}'

inferenceservice.ome.io/qwen2-5-7b patched

Pods stay Pending¶

List the scheduler's events in the namespace:

kubectl get events -n demo --field-selector reason=FailedScheduling

Their messages point at one of these causes:

• The InferenceService names a base model, and not enough nodes have it. Unless the model is served from a PVC, OME schedules the pods only onto nodes labeled models.ome.io/clusterbasemodel.qwen2-5-7b=Ready, or models.ome.io/demo.basemodel.qwen2-5-7b=Ready for a BaseModel. List them:

  ShellCopy

  kubectl get nodes -l models.ome.io/clusterbasemodel.qwen2-5-7b=Ready -o name

  Output

  node/gpu-node-1
  node/gpu-node-2

  An empty list means that no node has the model yet: go back to Step 2.

• The nodes lack free GPUs, CPU or memory for the pod's requests.

• The pods of a multi-pod Instance wait to be scheduled together: see Gang scheduling.

On OMENative, pods that stay unschedulable for too long fail their Instance: see Instances fail or never become ready.

Pods run, but the InferenceService isn't Ready¶

RawDeployment and MultiNode (deprecated) components need the Prometheus Operator's PodMonitor CRD. RawDeployment is the default for a component without a leader and workers. If the CRD is missing, their pods run, but the InferenceService never becomes Ready. OME records no event about it. Check for the CRD:

kubectl get crd podmonitors.monitoring.coreos.com

Error from server (NotFound): customresourcedefinitions.apiextensions.k8s.io "podmonitors.monitoring.coreos.com" not found

Install it as in The PodMonitor CRD, then restart the controller, which looks for the CRD only when it starts:

kubectl rollout restart deployment/ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

A change doesn't roll out¶

kubectl ome rollout explain shows the rollout plan and what holds it:

kubectl ome rollout explain qwen2-5-7b -n demo

Read the Effective view, the plan that OME follows:

kubectl ome rollout explain lists the other rows. Two other causes appear outside the plan:

• With spec.runtime.autoSync: false, the InferenceService pins its runtime, so runtime changes don't reach it. See Runtime revisions and pinning.
• A RetryHeld Warning event means that OME stopped retrying a failed update of an OMENative component. See Release a held revision.

The number of pods is wrong¶

kubectl ome autoscale status shows the autoscaling that the controller reports for each component:

kubectl ome autoscale status qwen2-5-7b -n demo

In a component's column, CLASS names the autoscaler, TARGET-NAME the object it scales, and CURRENT and DESIRED its replica counts. Add --live-scaler to also read the HorizontalPodAutoscaler or KEDA ScaledObject. Component autoscaling explains where the settings come from, and kubectl ome autoscale describes every row.

The InferenceService has no URL¶

If the InferenceService also lacks pods, start with The InferenceService gets no pods. If its pods run, see Pods run, but the InferenceService isn't Ready: a missing PodMonitor CRD also keeps OME from setting the URL.

The ome-resources chart turns off ingress creation by default. OME then points the URL at a Service named after the InferenceService, as in http://qwen2-5-7b.demo.svc.cluster.local:8080, and the IngressReady condition says so:

kubectl get inferenceservice qwen2-5-7b -n demo -o jsonpath='{.status.conditions[?(@.type=="IngressReady")].reason}'

IngressDisabled

OME skips the Service and the URL for an InferenceService labeled networking.knative.dev/visibility: cluster-local, and for one without an engine or a router. To reach the service from outside the cluster, see Expose a service without ingress or Configure ingress.

Requests fail¶

Send the InferenceService a test request, as in Step 7 of Serve your first model. If it fails, read the engine's logs, as in Step 4.

When the InferenceService sets a traffic policy, a traffic annotation or a canary, kubectl ome traffic explain compares it with what the controller reports. Suppose qwen2-5-7b is exposed through Envoy Gateway and sets spec.traffic.algorithm: RoundRobin:

kubectl ome traffic explain qwen2-5-7b -n demo

LAYER       STATE           VALUE           SOURCE
SUMMARY     Consistent      -               Computed/Current
INTENT      Declared        RoundRobin      Declared/Current
SUPPORT     Honored         -               Computed/Current
TRANSLATE   Computed        envoy-gateway   Computed/Current
REALIZE     Reported        r=1 e=0 w=0     Reported/Current
CHECK       Match           algorithm       Computed/Current
CHECK       Match           policy          Computed/Current
CHECK       NotApplicable   canary-weight   Computed/Current

SUMMARY is Consistent when the declared and reported policy agree, and kubectl ome traffic explains Mismatch and Unsupported. The command reads only the InferenceService, so Consistent doesn't prove that requests reach the pods. See Traffic policy.

Part of an Instance stays Pending¶

OME gang-schedules a multi-pod Instance through a PodGroup, which needs the scheduler-plugins PodGroup CRD. If the CRD is missing, OME creates the pods anyway, so the scheduler can place some of them, and sets GangSchedulingUnavailable to True on the component. Check its reason:

kubectl get inferenceservice qwen2-5-7b -n demo -o jsonpath='{.status.components.engine.lifecycle.conditions[?(@.type=="GangSchedulingUnavailable")].reason}'

PodGroupCRDNotInstalled

Install the CRD and restart the controller, as Install the PodGroup CRD shows. With the CRD in place, a MaybeNoGangScheduler Warning event means that the pods use the default scheduler, which ignores PodGroups unless you added the Coscheduling plugin. See Bring a gang-aware scheduler.

Instances fail or never become ready¶

When OME moves an Instance to the Failed phase, it records an InstanceFailed Warning event on the InferenceService. List these events:

kubectl get events -n demo --field-selector reason=InstanceFailed

The message names the Instance and its pod, and ends with one of these causes:

To find the failed Instances and read their last failures, follow Step 1 of Reset failed Instances.

Fix the cause. A fix to the pod template, such as a new image, rolls out to every Instance, failed ones included. After any other fix, release the held revision if the InferenceService has a RetryHeld Warning event, or else reset the failed Instances.

When an Instance stays unready but never fails, check the engine's deadline:

kubectl get inferenceservice qwen2-5-7b -n demo -o jsonpath='{.status.components.engine.lifecycle.conditions[?(@.type=="InstanceReadyTimeoutUnconfigured")].message}'

Instance readiness deadline is 30m0s

If the message starts with no instanceReadyTimeout is set, OME waits for you instead of failing the Instance. The Instance is stuck and never fails covers this and the other causes.

Pods stay Terminating after a delete¶

Deleting an InferenceService deletes its InferenceReplicas, and each keeps the finalizer ome.io/ir-teardown until its pods and PodGroups are gone. A pod on a node that stopped responding holds that up. OME records TeardownBlocked or TeardownDeadlineExceeded events on the InferenceReplica, not the InferenceService:

kubectl get events -n demo --field-selector involvedObject.kind=InferenceReplica,involvedObject.name=qwen2-5-7b-engine

Both messages end with a kubectl patch command that removes the finalizer. Read Recover stuck deletions before you run it.

Next steps¶

• kubectl ome status: every row of the status report.
• Troubleshoot runtime selection: when no runtime matches the model.
• Observe OMENative in status: the status that OMENative components report.