Runtime deployment-mode matching

When an InferenceService doesn't name a runtime, OME skips any serving runtime that declares a different deployment mode for the engine or decoder than the InferenceService asks for. Declare a mode on a runtime to keep it away from InferenceServices that ask for a different one. The catalog runtimes declare no mode, so they always pass this check.

The matching rule¶

OME compares the engine's modes, and the decoder's when the InferenceService has a spec.decoder. It doesn't compare the router. It rejects the runtime when both sides declare a mode for the same component and the modes differ. For the engine:

The check is a filter: it adds nothing to a runtime's score. OME checks the mode before the model format, so runtimes built for other models can show a mode rejection too. Troubleshoot runtime selection lists every check in order.

Where each side declares a mode¶

Both sides use the ome.io/deploymentMode annotation, and an InferenceService can also set spec.deploymentMode:

A mode counts when its value is RawDeployment, OMENative (since v1.3), MultiNode (deprecated) or VirtualDeployment (legacy). Annotation values are case-sensitive, and OME silently treats a misspelled one as no mode.

Since v1.3, when the engine or the decoder asks for OMENative, the webhook requires both to ask for the same mode. See For one component.

The runtime's mode also sets the running mode¶

OME merges the runtime's engineConfig, decoderConfig and routerConfig into the components the InferenceService declares, annotations included. So a runtime's mode becomes the component's mode, ahead of spec.deploymentMode, and only the component's own annotation in the InferenceService overrides it. An InferenceService that declares no mode can get a MultiNode runtime and run as MultiNode. Since v1.3, kubectl ome runtime effective shows each component's mode and the rule that set it. How OME resolves the mode gives the full order.

A runtime whose routerConfig declares MultiNode passes matching, but a router can't run in that mode. For an InferenceService that sets spec.router and gets that runtime, OME can't create or update the router. The manager logs Failed to reconcile component with invalid deployment mode for router, and records no event.

Example¶

This ClusterServingRuntime runs its engine in the deprecated MultiNode mode:

apiVersion: ome.io/v1beta1
kind: ClusterServingRuntime
metadata:
  name: srt-llama-3-3-70b-instruct-multinode
spec:
  supportedModelFormats:
    - modelFramework:
        name: transformers
        version: "4.47.0.dev0"
      modelFormat:
        name: safetensors
        version: "1.0.0"
      modelArchitecture: LlamaForCausalLM
      autoSelect: true
      priority: 2
  protocolVersions:
    - openAI
  modelSizeRange:
    min: 60B
    max: 75B
  engineConfig:
    annotations:
      ome.io/deploymentMode: MultiNode
    leader:
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
      runner:
        name: ome-container
        image: docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64
        ports:
          - containerPort: 8080
            name: http1
            protocol: TCP
        command:
          - python3
          - -m
          - sglang.launch_server
          - --host
          - "0.0.0.0"
          - --port
          - "8080"
          - --model-path
          - $(MODEL_PATH)
          - --tp-size
          - "8"
          - --nccl-init
          - $(LWS_LEADER_ADDRESS):5000
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --served-model-name
          - meta-llama/Llama-3.3-70B-Instruct
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm
        resources:
          requests:
            nvidia.com/gpu: 4
          limits:
            nvidia.com/gpu: 4
    worker:
      size: 1
      volumes:
        - name: dshm
          emptyDir:
            medium: Memory
      runner:
        name: ome-container
        image: docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64
        command:
          - python3
          - -m
          - sglang.launch_server
          - --host
          - "0.0.0.0"
          - --port
          - "8080"
          - --model-path
          - $(MODEL_PATH)
          - --tp-size
          - "8"
          - --nccl-init
          - $(LWS_LEADER_ADDRESS):5000
          - --nnodes
          - $(LWS_GROUP_SIZE)
          - --node-rank
          - $(LWS_WORKER_INDEX)
          - --served-model-name
          - meta-llama/Llama-3.3-70B-Instruct
        volumeMounts:
          - mountPath: /dev/shm
            name: dshm
        resources:
          requests:
            nvidia.com/gpu: 4
          limits:
            nvidia.com/gpu: 4

This InferenceService asks for OMENative for all its components:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-70b
  namespace: llama-demo
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-3-70b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

llama-3-3-70b-instruct is a ClusterBaseModel, the default kind. Auto-selection skips the runtime above, and any other runtime whose engine declares a mode other than OMENative.

Where a rejection shows up¶

OME applies the rule when it picks a runtime, and when it checks a runtime that the InferenceService names. A rejection gives this reason, with engine or decoder as the component and the runtime's mode first:

runtime engine deployment mode MultiNode does not match requested engine deployment mode OMENative

When the only runtimes are the example runtime and a ClusterServingRuntime with no autoSelect: true entry, the example InferenceService gets this message:

no runtime found to support model safetensors with format safetensors in namespace llama-demo. Checked 2 runtimes (0 namespace-scoped, 2 cluster-scoped). Excluded runtimes: srt-llama-3-3-70b-instruct-multinode (runtime engine deployment mode MultiNode does not match requested engine deployment mode OMENative)

The message names the model by its format, and leaves out the other runtime because it passed every check.

If the example InferenceService also sets spec.runtime.name: srt-llama-3-3-70b-instruct-multinode, the event is:

Runtime srt-llama-3-3-70b-instruct-multinode does not declare support for model llama-3-3-70b-instruct (runtime srt-llama-3-3-70b-instruct-multinode does not support model : runtime engine deployment mode MultiNode does not match requested engine deployment mode OMENative); proceeding because the runtime was named explicitly

Resolve a conflict¶

• Use a runtime that declares the InferenceService's mode, or none. To move a MultiNode runtime to OMENative, make an OMENative copy of it.
• Change the InferenceService's mode for that component to the runtime's, or remove it. Either way, the component runs in the runtime's mode.
• Name the runtime in spec.runtime.name. OME uses it with a warning, and the component runs in the runtime's mode unless the component's own annotation sets one.
• To keep InferenceServices that declare no mode off a runtime, leave autoSelect unset or false on all its entries. Then name the runtime where you need it.

Related pages¶

• Deployment modes and OMENative: how each mode runs a component.
• Move from LeaderWorkerSet to OMENative: move a MultiNode component and its runtime to OMENative.
• Troubleshoot runtime selection: find out why OME picked a runtime, or none.
• Reference a runtime explicitly: name the runtime instead of letting OME pick one.
• kubectl ome runtime: see why a runtime was skipped, and each component's mode.
• Runtime accelerator-class matching: how accelerator classes narrow the runtimes OME can pick.
• Model version matching: how a model's format and framework versions match a runtime's entries.
• OME API: the fields of ServingRuntimeSpec, including engineConfig and decoderConfig.