Deploy an InferenceService

An InferenceService is the one resource you have to write to serve a model. At its smallest it names a model. OME picks a runtime that supports the model, creates the serving pods and a Service, and reports when the service is ready. This guide writes the three smallest shapes an InferenceService takes, then follows one of them from kubectl apply to a request. The other guides in this section, accelerators, storage and a runtime you name, are variations on it.

Step 1: Choose the shape¶

An InferenceService names a model, a runtime, or both. Pick the shape that fits, and save it as isvc.yaml.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine: {}

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: vllm-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-runtime-only
  namespace: llama-demo
spec:
  runtime:
    name: vllm-llama-3-2-1b-instruct
  engine:
    runner:
      env:
        - name: MODEL_PATH
          value: meta-llama/Llama-3.2-1B-Instruct
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

The webhook rejects an InferenceService that names neither: at least one of spec.model or spec.runtime must be set.

The rest of this guide uses the first shape. The commands work for the second one unchanged. For the third, use the name llama-runtime-only in place of llama-3-2-1b-instruct.

Step 2: Apply it¶

kubectl apply -f isvc.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-3-2-1b-instruct created

The warning names the runtime OME picked. When you name a runtime that declares support for the model, there's no warning. If no runtime that allows automatic selection supports the model, the webhook rejects the InferenceService with a message that starts with no supporting runtime found for model llama-3-2-1b-instruct: see Troubleshoot runtime selection.

Step 3: Wait until it's ready¶

The first start pulls the runtime's image and loads the model, which can take several minutes:

kubectl wait --for=condition=Ready inferenceservice/llama-3-2-1b-instruct -n llama-demo --timeout=30m

inferenceservice.ome.io/llama-3-2-1b-instruct condition met

status.url is where to send requests. With ingress creation off, the default, it's the address of the Service that OME puts in front of the model:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo -o jsonpath='{.status.url}'

http://llama-3-2-1b-instruct.llama-demo.svc.cluster.local:8080

Step 4: See what OME created¶

With the chart's default settings, OME creates:

• A Deployment and a Service named llama-3-2-1b-instruct-engine. The pods run only on nodes where the model is Ready, as Node labels and status describes.
• A HorizontalPodAutoscaler and a PodDisruptionBudget of the same name. The autoscaler keeps CPU use at 80% with 1 to 3 replicas, and the budget lets one pod be unavailable at a time.
• The Service llama-3-2-1b-instruct in front of the engine, whose address is the URL above. See Ingress and external access.

The RUNTIME column of kubectl get inferenceservice shows spec.runtime.name, so it's empty when OME picks the runtime. The engine's workload always names the runtime OME used, in its serving-runtime label:

kubectl get deployment llama-3-2-1b-instruct-engine -n llama-demo -o jsonpath='{.metadata.labels.serving-runtime}'

vllm-llama-3-2-1b-instruct

The engine runs as a Deployment because nothing chose another deployment mode. A multi-node engine runs on OMENative by default, and any component can opt in.

Step 5: Send a request¶

The Service is a ClusterIP Service, so forward a local port to it, and leave the command running:

kubectl port-forward svc/llama-3-2-1b-instruct 8080:8080 -n llama-demo

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

In a second terminal, ask the runtime which model names it serves:

curl -s http://localhost:8080/v1/models

The response lists them under data. Send a chat completion to /v1/chat/completions with one of those names as model, as Step 7 of Serve your first model does. The response is an OpenAI chat completion, with the answer in choices[0].message.content.

Troubleshooting¶

Troubleshoot an InferenceService covers more symptoms.

The webhook rejects the InferenceService¶

kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, then a message:

The engine pod stays Pending¶

Look at the pod's events with kubectl describe pod -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct. Insufficient nvidia.com/gpu means no node has a free GPU for the runtime's request. didn't match Pod's node affinity/selector means the model isn't Ready on any node with a free GPU: list the nodes that have it with kubectl get nodes -l models.ome.io/clusterbasemodel.llama-3-2-1b-instruct=Ready.

The engine pod runs, but the InferenceService doesn't become ReadyUnreleased: coming in v1.3¶

Engines that run as a Deployment need the Prometheus Operator's PodMonitor CRD. Install it as in The PodMonitor CRD, then restart the controller with kubectl rollout restart deployment/ome-controller-manager -n ome, because it looks for the CRD only when it starts.

Clean up¶

Delete the InferenceService. The model and the runtime stay, for other InferenceServices to use:

kubectl delete inferenceservice llama-3-2-1b-instruct -n llama-demo

inferenceservice.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace

Next steps¶

• Select accelerators: name an AcceleratorClass for the engine, or let a policy pick one.
• Serve models from a PVC and Serve models from node-local storage: serve weights that are already in the cluster, with no download.
• Reference a runtime explicitly: the fields of spec.runtime, and how OME resolves the name.
• Serve a model on OMENative: run the engine on OMENative instead of a Deployment. Since v1.3.
• InferenceService: the engine, decoder and router components, and every field they take.