Serve a model on OMENative Unreleased: coming in v1.3

Opt a single-pod InferenceService into OMENative with spec.deploymentMode, then check the InferenceReplica, the Instance and the pod that OME creates.

Run a single-pod model on OMENative, OME's own workload controller, by setting spec.deploymentMode: OMENative on its InferenceService. OMENative runs each replica as an Instance and reports its phase and last failure. It also stops retrying an image that can't be pulled, and lets you choose how an update replaces the pod.

An engine or decoder with a leader or worker runs on OMENative by default, and a single-pod component runs as a Deployment until you opt it in. Here you opt in the model from Serve your first model, check what OME creates for it, and send it a request.

Step 1: Create the InferenceService¶

Create a namespace for the InferenceService:

Shell

kubectl create namespace qwen3-native

Output

namespace/qwen3-native created

Save this InferenceService as qwen3-0-6b-native.yaml. It's the one from Serve your first model, with the new namespace and spec.deploymentMode:

qwen3-0-6b-native.yaml

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-native
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 1
    maxReplicas: 1

deploymentMode: OMENative runs every component of the InferenceService on OMENative, here only the engine. minReplicas and maxReplicas count Instances, not pods. An Instance of this engine is one pod, so the engine runs one pod.

Create the InferenceService:

Shell

kubectl apply -f qwen3-0-6b-native.yaml

Output

inferenceservice.ome.io/qwen3-0-6b created

Wait for the InferenceService to be ready. Pulling the SGLang image and loading the model can take several minutes:

Shell

kubectl wait --for=condition=Ready inferenceservice/qwen3-0-6b -n qwen3-native --timeout=30m

Output

inferenceservice.ome.io/qwen3-0-6b condition met

Step 2: Check the InferenceReplica¶

OME runs an OMENative component from an InferenceReplica named {isvc}-{component}, in place of a Deployment. List the InferenceReplicas, irep for short:

Shell

kubectl get inferencereplicas -n qwen3-native

Output

NAME                COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
qwen3-0-6b-engine   engine      1         1         1       1           6m

The columns count Instances. DESIRED is the number that OME keeps, here the engine's minReplicas, and READY counts the Instances whose pods are all ready. Readiness and availability explains AVAILABLE.

No Deployment runs the engine:

Shell

kubectl get deployments -n qwen3-native

Output

No resources found in qwen3-native namespace.

Make your changes in the InferenceService. OME keeps the InferenceReplica's spec in line with the InferenceService and its runtime.

Step 3: Check the Instance and its pod¶

List the Instances of the InferenceService with kubectl ome:

Shell

kubectl ome instance list qwen3-0-6b -n qwen3-native

Output

COMP     IDX/INC   PHASE   PODS    REVS          AOF   EVIDENCE
engine   0/1       Ready   1/1/1   qwen...9c7a   A--   OK

IDX/INC is 0/1: Instance 0, in its first incarnation.
PHASE is Ready. While the pod starts, it's Creating.
PODS counts the Instance's serving, available and total pods.

Output fields explains the other columns, and Instance phases describes the phases.

OME names the pod {isvc}-{component}-{index}-{runner}-{ordinal}, where the runner of a single-pod Instance is default. List it with its ome.io/serving condition:

Shell

kubectl get pods -n qwen3-native -l ome.io/inferenceservice=qwen3-0-6b \
  -o 'custom-columns=NAME:.metadata.name,SERVING:.status.conditions[?(@.type=="ome.io/serving")].status'

Output

NAME                            SERVING
qwen3-0-6b-engine-0-default-0   True

ome.io/serving is a readiness gate that OME controls. The pod is ready, and receives requests, only while it's True. OME sets it to False to take the pod out of rotation without stopping it, for example to drain it. Pod labels and environment lists the labels and OME_* environment variables that OME adds to the pod.

Step 4: Send a request¶

Forward a local port to the Service qwen3-0-6b, as in Serve your first model, and leave the command running. If you turned on ingress creation, forward to svc/qwen3-0-6b-engine instead.

Shell

kubectl port-forward svc/qwen3-0-6b 8080:8080 -n qwen3-native

Output

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

In a second terminal, send the chat completion request from Serve your first model. chat_template_kwargs turns off Qwen3's thinking mode, which can use up max_tokens:

Shell

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is Kubernetes? Answer in one sentence."}], "max_tokens": 100, "chat_template_kwargs": {"enable_thinking": false}}'

The response is an OpenAI chat completion, with the model's answer in choices[0].message.content.

Move a running InferenceService to OMENative¶

To move an InferenceService that runs as a Deployment, such as the one from Serve your first model, add deploymentMode: OMENative to its spec and apply it again. OME leaves the old Deployment running. Until you delete it with kubectl delete deployment qwen3-0-6b-engine -n qwen3-0-6b, its pod keeps its GPU, and the new pod can stay Pending.

Troubleshooting¶

The pod stays Pending¶

Read the scheduler's message in the pod's events with kubectl describe pod -n qwen3-native -l ome.io/inferenceservice=qwen3-0-6b. The reasons are the ones in The engine pod stays Pending, in Serve your first model. Also check whether the InferenceService from that guide, or a Deployment left from a mode change, still holds the GPU.

The Instance fails¶

When kubectl ome instance list shows the phase Failed, run kubectl ome instance status qwen3-0-6b 0 --component engine -n qwen3-native. It shows the Instance's last failure, its pods and their warning events. Read SGLang's logs with kubectl logs -n qwen3-native -l ome.io/inferenceservice=qwen3-0-6b --tail=100.

With the chart's settings, OME fails a new Instance when:

a container of its pod is stuck, such as in ImagePullBackOff or CrashLoopBackOff, and the pod is more than 60 seconds old;
its pod stays unschedulable for 15 minutes;
it takes more than 30 minutes to become ready, excluding any time its pod spends unschedulable.

Set Instance readiness deadlines explains how to change these deadlines.

OME can rebuild a failed Instance on its own. When the image can't be pulled, the chart's settings give the revision three attempts in all. Then OME holds the revision and stops rebuilding the Instance. What OME does with a failed Instance describes the retries.

If your fix changes the pod template, such as the image, OME rolls the new revision out to every Instance of the engine, the failed one included. For any other fix, release the held revision if OME holds it, and otherwise reset the failed Instance.

The engine runs as a Deployment¶

If kubectl get deployments -n qwen3-native lists qwen3-0-6b-engine and there's no InferenceReplica, look for an ome.io/deploymentMode annotation on the engine: it wins over spec.deploymentMode. Look in the InferenceService with kubectl get inferenceservice qwen3-0-6b -n qwen3-native -o jsonpath='{.spec.engine.annotations}'. Look in the runtime with kubectl get clusterservingruntime srt-qwen3-0-6b -o jsonpath='{.spec.engineConfig.annotations}'.

Set ome.io/deploymentMode: OMENative in the InferenceService's spec.engine.annotations, which wins over the runtime's annotation. Then delete the Deployment that OME leaves behind with kubectl delete deployment qwen3-0-6b-engine -n qwen3-native. How OME resolves the mode gives the full order.

No pod appears¶

If the InferenceReplica exists but its CURRENT, READY and AVAILABLE columns stay empty, check that the manager runs the InferenceReplica controller. Read the startup logs of every manager replica:

Shell

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'InferenceReplica controller'

A replica that runs the controller logs Setting up InferenceReplica controller. If one logs InferenceReplica controller disabled via --enable-inferencereplica-controller=false, the manager container of the ome-controller-manager Deployment has that flag. The chart doesn't set it, so remove it where it was added.

The webhook rejects the InferenceService¶

The webhook rejects the InferenceService when the model or the runtime doesn't exist, or when the model has no recorded format. The InferenceService is rejected or gets no pods, in Serve your first model, lists the messages and their fixes.

Clean up¶

Delete the InferenceService and its namespace:

Shell

kubectl delete inferenceservice qwen3-0-6b -n qwen3-native
kubectl delete namespace qwen3-native

Output

inferenceservice.ome.io "qwen3-0-6b" deleted from qwen3-native namespace
namespace "qwen3-native" deleted

Deleting the namespace waits until OME has drained and deleted the pod. If a pod stays Terminating, such as on a node that's gone, see Recover stuck deletions.

The model and the runtime stay. To delete them too, follow Clean up in Serve your first model.

Next steps¶

Serve a multi-node model: give the engine a leader and workers, which OMENative runs as multi-pod Instances.
OMENative overview: what OMENative does, and how its parts fit together.
Instances: the phases, operations and incarnations of an Instance.
OMENative update strategies: how OME replaces an Instance's pods when you change the InferenceService. The default needs room for a second pod, so with one free GPU, choose RecreatePod.
Deployment modes and OMENative: the modes compared, and everything OME creates in each.
kubectl ome instance: inspect Instances and their failures.