Serve your first model

Create a ClusterBaseModel, a ClusterServingRuntime and an InferenceService for a small model, then send your first request to it.

Serve Qwen3-0.6B, a small language model, with SGLang on one GPU, and send it a chat request through its OpenAI-compatible API. You create a ClusterBaseModel that says where the weights are, a ClusterServingRuntime that describes the serving pods, and an InferenceService that serves the model with that runtime.

Step 1: Check that OME is running¶

Check that the OME controller and the model agent are ready:

Shell

kubectl wait --for=condition=Available deployment/ome-controller-manager -n ome --timeout=5m
kubectl rollout status daemonset/ome-model-agent-daemonset -n ome --timeout=5m

Output

deployment.apps/ome-controller-manager condition met
daemon set "ome-model-agent-daemonset" successfully rolled out

If the second command fails with NotFound, run Step 3 of Install OME again: its values file turns on the model agent.

Step 2: Create the model¶

A ClusterBaseModel is cluster-scoped, so InferenceServices in any namespace can serve it. Save this one as qwen3-0-6b-model.yaml:

qwen3-0-6b-model.yaml

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: qwen3-0-6b
spec:
  storage:
    storageUri: hf://Qwen/Qwen3-0.6B
    path: /mnt/data/models/Qwen/Qwen3-0.6B

storageUri says where the weights come from: hf://, then the model's repository on Hugging Face.
path is the directory on each node that holds the weights. Keep it inside the model agent's host directory, /mnt/data/models by default, so that the weights land on the node's disk.

Every node that runs the model agent downloads the model. To keep it on some nodes only, see Choose the nodes.

Create the model:

Shell

kubectl apply -f qwen3-0-6b-model.yaml

Output

clusterbasemodel.ome.io/qwen3-0-6b created

Step 3: Watch the model become Ready¶

Watch the model's state:

Shell

kubectl get clusterbasemodel qwen3-0-6b -w -o custom-columns=NAME:.metadata.name,STATE:.status.state

Output

NAME         STATE
qwen3-0-6b   In_Transit
qwen3-0-6b   In_Transit
qwen3-0-6b   Ready

Press Ctrl+C once the state is Ready. <none> means that OME hasn't set a state yet.

Meanwhile, the model agent on each node downloads the weights into /mnt/data/models/Qwen/Qwen3-0.6B. The model is Ready as soon as one node has it. OME reads the model's format and architecture from its files, and fills them in for you. Check what OME learned:

Shell

kubectl get clusterbasemodel qwen3-0-6b \
  -o custom-columns=NAME:.metadata.name,FORMAT:.spec.modelFormat.name,ARCHITECTURE:.spec.modelArchitecture,STATE:.status.state

Output

NAME         FORMAT        ARCHITECTURE       STATE
qwen3-0-6b   safetensors   Qwen3ForCausalLM   Ready

Step 4: Create the runtime¶

The ClusterServingRuntime srt-qwen3-0-6b, from OME's catalog, runs SGLang from the image docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64, which is built for amd64 and CUDA 12.9. SGLang serves the model on port 8080, under the name Qwen/Qwen3-0.6B. The runtime sets autoSelect: false, so an InferenceService must name it to use it.

Apply the runtime from the v1.2.2 release:

Shell

kubectl apply -f https://raw.githubusercontent.com/ome-projects/ome/v1.2.2/config/runtimes/srt/Qwen/qwen3-0-6b-rt.yaml

Output

clusterservingruntime.ome.io/srt-qwen3-0-6b created

Step 5: Create the InferenceService¶

Create a namespace for the InferenceService:

Shell

kubectl create namespace qwen3-0-6b

Output

namespace/qwen3-0-6b created

Save this InferenceService as qwen3-0-6b-isvc.yaml:

qwen3-0-6b-isvc.yaml

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-0-6b
  namespace: qwen3-0-6b
spec:
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-0-6b
  engine:
    minReplicas: 1
    maxReplicas: 1

model.name names the model from Step 2, and model.kind defaults to ClusterBaseModel. To serve a BaseModel from the InferenceService's namespace instead, set model.kind: BaseModel.
runtime.name names the runtime from Step 4.
engine is the component that runs the model. The runtime supplies its container, and minReplicas and maxReplicas keep it at one pod.

Create the InferenceService:

Shell

kubectl apply -f qwen3-0-6b-isvc.yaml

Output

inferenceservice.ome.io/qwen3-0-6b created

Step 6: Watch the InferenceService become Ready¶

Wait for the InferenceService to be ready. The first start pulls the SGLang image, which can take several minutes:

Shell

kubectl wait --for=condition=Ready inferenceservice/qwen3-0-6b -n qwen3-0-6b --timeout=30m

Output

inferenceservice.ome.io/qwen3-0-6b condition met

Then look at the InferenceService:

Shell

kubectl get inferenceservice qwen3-0-6b -n qwen3-0-6b

Output

NAME         URL                                                   READY   BASEMODEL    RUNTIME          AGE
qwen3-0-6b   http://qwen3-0-6b.qwen3-0-6b.svc.cluster.local:8080   True    qwen3-0-6b   srt-qwen3-0-6b   7m12s

The chart turns off ingress creation by default, so OME puts the Service qwen3-0-6b in front of the engine, and URL is its address inside the cluster.

OME runs the engine as the Deployment qwen3-0-6b-engine, in the default RawDeployment deployment mode. Its pod runs on a node where the model is Ready, and mounts the weights from that node. List it:

Shell

kubectl get pods -n qwen3-0-6b

Output

NAME                                 READY   STATUS    RESTARTS   AGE
qwen3-0-6b-engine-6d8f9c7b5d-x2k4p   1/1     Running   0          7m8s

Step 7: Send a request¶

The Service qwen3-0-6b is a ClusterIP Service, so forward a local port to it, and leave the command running. If you turned on ingress creation, forward to svc/qwen3-0-6b-engine instead.

Shell

kubectl port-forward svc/qwen3-0-6b 8080:8080 -n qwen3-0-6b

Output

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

In a second terminal, send a chat completion request. model is the name that the runtime serves the model under. chat_template_kwargs turns off Qwen3's thinking mode, which can use up max_tokens on <think> reasoning before the answer.

Shell

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-0.6B", "messages": [{"role": "user", "content": "What is Kubernetes? Answer in one sentence."}], "max_tokens": 100, "chat_template_kwargs": {"enable_thinking": false}}'

The response is an OpenAI chat completion, with the model's answer in choices[0].message.content. If finish_reason is length, the answer ran out of tokens: raise max_tokens.

Troubleshooting¶

Troubleshoot an InferenceService covers more symptoms.

The model doesn't become Ready¶

Check the model's label on each node:

Shell

kubectl get nodes -L models.ome.io/clusterbasemodel.qwen3-0-6b

Output

NAME         STATUS   ROLES    AGE   VERSION   CLUSTERBASEMODEL.QWEN3-0-6B
gpu-node-1   Ready    <none>   12d   v1.33.4   Failed

Empty: no model agent runs on the node. List the agent's pods and their nodes with kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset -o wide. The agent tolerates the nvidia.com/gpu taint. For nodes with other taints, add tolerations to modelAgent.tolerations.
Updating: the agent is still downloading or reading the model.
Failed: the download or the reading failed, for example because the node can't reach Hugging Face.

For Updating and Failed, read the agent's logs with kubectl logs -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset --tail=100. Run the model agent covers the agent's settings.

The InferenceService is rejected or gets no pods¶

When the webhook rejects the InferenceService, kubectl's error includes admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, followed by a reason:

The reason includes	Fix
`referenced model "qwen3-0-6b" not found`	Create the model, as in Step 2.
`model format name is required`	Wait for the model to be `Ready`, as in Step 3.
`runtime srt-qwen3-0-6b not found`	Create the runtime, as in Step 4.

Then apply qwen3-0-6b-isvc.yaml again.

On v1.2.2, the webhook admits the InferenceService in the last two cases. OME creates no pods for it, and records the reason in RuntimeValidationError events:

Shell

kubectl get events -n qwen3-0-6b --field-selector reason=RuntimeValidationError

Output

LAST SEEN   TYPE      REASON                   OBJECT                        MESSAGE
8s          Warning   RuntimeValidationError   inferenceservice/qwen3-0-6b   Runtime srt-qwen3-0-6b does not support model qwen3-0-6b: runtime srt-qwen3-0-6b not found in namespace qwen3-0-6b or at cluster scope

Fix the cause, and OME creates the engine on its own.

The engine pod stays Pending¶

Look at the pod's events with kubectl describe pod -n qwen3-0-6b -l ome.io/inferenceservice=qwen3-0-6b. The scheduler's message says why no node fits:

Insufficient nvidia.com/gpu, Insufficient cpu or Insufficient memory: no node has a free GPU, 10 CPUs and 30 GiB of memory. On the GPU node, the model agent's pod already takes 10 CPUs and 100 GiB. Lower them with modelAgent.resources, or free the GPU.
didn't match Pod's node affinity/selector: the model isn't Ready on any node with a free GPU. List the nodes that have it with kubectl get nodes -l models.ome.io/clusterbasemodel.qwen3-0-6b=Ready.
untolerated taint: the GPU node has a taint other than nvidia.com/gpu, the only one that the runtime tolerates.

The engine pod isn't ready or restarts¶

The runtime's startup probe gives SGLang about 16 minutes to load the model before Kubernetes restarts its container. Read SGLang's logs with kubectl logs -n qwen3-0-6b -l ome.io/inferenceservice=qwen3-0-6b --tail=100.

ErrImagePull or ImagePullBackOff: the node can't pull the SGLang image from Docker Hub. If your nodes pull from a mirror, see Install from a private registry.
SGLang can't find the model's files: check that the model's path is inside the model agent's host directory. With a path outside it, the model still becomes Ready, but the engine's pod mounts an empty directory.

The engine pod runs, but the InferenceService doesn't become ReadyUnreleased: coming in v1.3¶

RawDeployment engines like this one need the Prometheus Operator's PodMonitor CRD. Install it as in The PodMonitor CRD. The controller looks for the CRD only when it starts, so then restart it with kubectl rollout restart deployment/ome-controller-manager -n ome.

Clean up¶

Serve a model on OMENative starts from this model and runtime. If you go on to it, delete only the InferenceService, which frees its GPU. To delete everything:

Shell

kubectl delete inferenceservice qwen3-0-6b -n qwen3-0-6b
kubectl delete namespace qwen3-0-6b
kubectl delete clusterservingruntime srt-qwen3-0-6b
kubectl delete clusterbasemodel qwen3-0-6b

Output

inferenceservice.ome.io "qwen3-0-6b" deleted from qwen3-0-6b namespace
namespace "qwen3-0-6b" deleted
clusterservingruntime.ome.io "srt-qwen3-0-6b" deleted
clusterbasemodel.ome.io "qwen3-0-6b" deleted

Deleting the model waits until the model agent has deleted /mnt/data/models/Qwen/Qwen3-0.6B from each node that has it. To keep the files, see Deleting a model.

Next steps¶

Serve a model on OMENative: run this model's engine on OMENative, OME's own workload controller, instead of a Deployment. Since v1.3.
Expose a service without ingress: reach the model from outside the cluster through a LoadBalancer or NodePort Service.
Pre-configured models and runtimes: serve more models with the ome-serving chart. Its qwen3-0-6b entry creates resources with the same names as yours, so clean up first if you turn it on.
Serving runtimes: how a runtime describes the pods that serve a model, and how OME picks one.
InferenceService: the engine, decoder and router components, and what OME creates for each.