Reference a runtime explicitly

When an InferenceService names a runtime in spec.runtime.name, OME serves it with that runtime instead of choosing one. Name a runtime when:

• OME wouldn't choose it on its own, for example because it sets autoSelect: false.
• Several runtimes match the model, and you want a particular one.
• You want the choice to stay fixed, because a new or changed runtime can replace one that OME chose.
• Since v1.3, the runtime inherits from a parent runtime. OME uses the inherited settings only for InferenceServices that name the runtime.
• Since v1.3, the runtime loads the weights itself, so the InferenceService needs no model. See Step 2.

OME uses the runtime you name even when the runtime doesn't declare support for the model, and warns you.

The steps below serve the ClusterBaseModel llama-3-2-1b-instruct with the ClusterServingRuntime srt-llama-3-2-1b-instruct, then check which runtime OME used.

Step 1: Check that the runtime supports the model¶

OME uses the runtime without a warning when:

• An entry in the runtime's supportedModelFormats has the model's architecture, format, framework and quantization, at matching versions.
• The model's spec.modelParameterSize, when OME has recorded it, is inside the runtime's modelSizeRange.
• The runtime accepts the InferenceService's accelerator class and deployment mode.

Supported model formats has the full rules.

srt-llama-3-2-1b-instruct sets autoSelect: false, so OME never picks it on its own, and you have to name it. Since v1.3, kubectl ome runtime explain lists the runtimes that OME considers for a model, whether it can choose each one, and why.

Compare what OME recorded for the model with what the runtime declares. The commands leave out the format version, which is 1.0.0 on both sides:

kubectl get clusterbasemodel llama-3-2-1b-instruct \
  -o custom-columns=ARCHITECTURE:.spec.modelArchitecture,FORMAT:.spec.modelFormat.name,FRAMEWORK:.spec.modelFramework.name,VERSION:.spec.modelFramework.version

ARCHITECTURE       FORMAT        FRAMEWORK      VERSION
LlamaForCausalLM   safetensors   transformers   4.45.0.dev0

kubectl get clusterservingruntime srt-llama-3-2-1b-instruct \
  -o 'custom-columns=NAME:.metadata.name,DISABLED:.spec.disabled,ARCHITECTURE:.spec.supportedModelFormats[*].modelArchitecture,FORMAT:.spec.supportedModelFormats[*].modelFormat.name,FRAMEWORK:.spec.supportedModelFormats[*].modelFramework.name,VERSION:.spec.supportedModelFormats[*].modelFramework.version,MIN:.spec.modelSizeRange.min,MAX:.spec.modelSizeRange.max'

NAME                        DISABLED   ARCHITECTURE       FORMAT        FRAMEWORK      VERSION       MIN    MAX
srt-llama-3-2-1b-instruct   false      LlamaForCausalLM   safetensors   transformers   4.45.0.dev0   500M   2B

The two match. To compare other runtimes, leave out the runtime's name, or use servingruntimes -n llama-demo for the ServingRuntimes in the namespace.

Step 2: Name the runtime in the InferenceService¶

Set spec.runtime.name to the runtime's name:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

To serve a BaseModel in llama-demo instead, add kind: BaseModel under spec.model, and run Step 1's first command with basemodel -n llama-demo. To move an InferenceService that already exists to the runtime, add the same runtime field to its manifest.

spec.runtime also takes these fields:

Since v1.3, you can leave out spec.model when the runtime loads the weights itself, from its container or from a volume or init container under spec.engine. OME then skips all model work, and sets no MODEL_PATH. A runtime that passes $(MODEL_PATH) to the engine, as this one does, then needs the variable in its runner's env. See The runtime.

Apply the InferenceService:

kubectl apply -f isvc.yaml

inferenceservice.ome.io/llama-3-2-1b-instruct created

Wait for the InferenceService to become ready:

kubectl wait --for=condition=Ready inferenceservice/llama-3-2-1b-instruct -n llama-demo --timeout=30m

inferenceservice.ome.io/llama-3-2-1b-instruct condition met

Step 3: Check the runtime OME used¶

The RUNTIME column of kubectl get inferenceservice shows the name you asked for. To see the runtime that OME built the serving pods from, read their serving-runtime label:

kubectl get pods -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct \
  -o custom-columns=COMPONENT:.metadata.labels.component,RUNTIME:.metadata.labels.serving-runtime

COMPONENT   RUNTIME
engine      srt-llama-3-2-1b-instruct

The label holds the runtime's name, not its kind. Since v1.3, kubectl ome runtime effective llama-3-2-1b-instruct -n llama-demo shows the kind too, as CSR/srt-llama-3-2-1b-instruct. It also shows the pin and drift, and with -o json or -o yaml, the runtime selection and inheritance. See kubectl ome runtime.

Last, check for RuntimeCompatibilityAdvisory events, which OME records when the runtime doesn't declare support for the model. This runtime does, so there are none:

kubectl get events -n llama-demo --field-selector involvedObject.name=llama-3-2-1b-instruct,reason=RuntimeCompatibilityAdvisory

No resources found in llama-demo namespace.

How OME resolves the name¶

OME looks for the name in the InferenceService's namespace and at cluster scope. Since v1.3, spec.runtime.kind decides the order:

When a ClusterServingRuntime and a ServingRuntime in the namespace share a name, set kind: ServingRuntime to get the ServingRuntime. On v1.2.2, OME ignores kind: it looks for a ServingRuntime in the namespace first, then for a ClusterServingRuntime.

Troubleshooting¶

Since v1.3, the admission webhook checks the runtime and the model when you create or update an InferenceService that has an engine section. On v1.2.2, or without an engine section, the same problems show up as events after kubectl apply succeeds.

kubectl prints a does not declare support warning¶

The runtime doesn't declare support for the model, or doesn't accept the InferenceService's accelerator class or deployment mode. The text in parentheses says what differs. For example, if the model's config.json had "transformers_version": "4.46.0", OME would record the framework version 4.46.0, while this runtime declares 4.45.0.dev0:

kubectl apply -f isvc.yaml

Warning: runtime "srt-llama-3-2-1b-instruct" does not declare support for model "llama-3-2-1b-instruct" (runtime srt-llama-3-2-1b-instruct does not support model safetensors: model format 'mt:safetensors:1.0.0:LlamaForCausalLM:transformers:4.46.0' not in supported formats: framework version mismatch (model=4.46.0, runtime=4.45.0.dev0)); proceeding because the runtime was named explicitly
inferenceservice.ome.io/llama-3-2-1b-instruct created

OME serves the model with the runtime anyway. If the runtime can't load the model, the serving pods don't become ready: name a runtime that declares the model instead. While the InferenceService follows the live runtime, OME also records a RuntimeCompatibilityAdvisory event with the same details, which the last command in Step 3 lists.

The webhook rejects the InferenceService¶

kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, then a message:

The webhook rejects the InferenceService lists other messages.

The runtime was deleted or disabled¶

Since v1.3, if you delete the runtime while the InferenceService follows it, OME records a RuntimeNotFound event and sets the RuntimeReady condition to False. It stops updating the serving workload, but running pods keep serving. RuntimeReady doesn't count toward Ready, so the InferenceService stays Ready while they serve:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o 'custom-columns=STATUS:.status.conditions[?(@.type=="RuntimeReady")].status,REASON:.status.conditions[?(@.type=="RuntimeReady")].reason,MESSAGE:.status.conditions[?(@.type=="RuntimeReady")].message'

STATUS   REASON            MESSAGE
False    RuntimeNotFound   runtime srt-llama-3-2-1b-instruct not found in namespace llama-demo or at cluster scope

To recover, recreate the runtime, or point spec.runtime.name at one that exists. OME then resumes updating the workload, and sets RuntimeReady to True with the reason RuntimeResolved. Until then, the webhook rejects every update to the InferenceService, even a label change, unless the update names a runtime that exists. You can still delete the InferenceService.

With autoSync: false, the InferenceService keeps serving from its pinned snapshot instead, and its RuntimeDrifted condition turns True with the reason SourceRuntimeMissing. See The RuntimeDrifted condition.

On v1.2.2, a deleted runtime gives a RuntimeValidationError event, or RuntimePinError with autoSync: false.

If you disable a runtime that the InferenceService follows, OME records a RuntimeValidationError event that says the runtime is disabled, and stops updating the serving workload. Running pods keep serving. To recover, set disabled: false, or name another runtime.

Clean up¶

Delete the InferenceService:

kubectl delete inferenceservice llama-3-2-1b-instruct -n llama-demo

inferenceservice.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace

The model and the runtime stay, for other InferenceServices to use.

Next steps¶

• Troubleshoot runtime selection: find out why OME passes over a runtime when it selects one itself.
• Serving runtimes: what a runtime declares, and how OME builds pods from it.
• Runtime revisions and pinning: pin an InferenceService to a snapshot of its runtime.
• Runtime inheritance: keep the settings that runtimes share in one parent runtime.