Runtime selection scoring

When an InferenceService names a model but no runtime, OME picks one of the serving runtimes that support the model. A ServingRuntime in the InferenceService's namespace comes before any ClusterServingRuntime, and among runtimes of the same kind the highest score wins: (format weight + framework weight) × priority. To prefer a runtime, raise its entry's priority. To fix the choice, name the runtime.

The ranking order¶

OME ranks the runtimes that qualify for the model. A runtime qualifies when:

• it isn't disabled
• it passes the accelerator-class and deployment-mode checks
• one of its supportedModelFormats entries matches the model, and the model's size fits its modelSizeRange
• one of its entries sets autoSelect: true
• its score is above 0

OME picks the first runtime in this order:

1. ServingRuntimes in the InferenceService's namespace come before ClusterServingRuntimes, whatever their scores.
2. A higher score comes first.
3. For a model with a modelParameterSize, a size range closer to the model's size comes first.
4. Runtimes that still tie come in name order.

Troubleshoot runtime selection shows what you see when no runtime qualifies.

Steer the choice¶

To make OME pick the runtime you want:

• Raise the priority or the weights of the runtime's entry for the model's format and framework.
• Override the cluster's runtimes in one namespace with a ServingRuntime there that supports the model and sets autoSelect: true on an entry.
• Name the runtime in spec.runtime.name. OME then uses it, so the choice holds when you add or change other runtimes. See Reference a runtime explicitly.

When OME picks a runtime, it matches and renders the runtime's own spec, without inheritance, so name a child runtime instead of setting autoSelect: true on it.

How the score is computed¶

A runtime's score is the highest score among its supportedModelFormats entries. An entry that matches the model's format and framework, versions included, scores (modelFormat.weight + modelFramework.weight) × priority:

Writing weight: 0 raises the score rather than switching the weight off. A weight can be negative, which lowers the score.

This entry, from vllm-llama in the worked example, scores (2 + 1) × 2 = 6:

spec:
  supportedModelFormats:
    - modelFormat:
        name: safetensors
        version: "1.0.0"
        weight: 2
      modelFramework:
        name: transformers
        version: "4.45.0.dev0"
      modelArchitecture: LlamaForCausalLM
      autoSelect: true
      priority: 2

The score ignores modelArchitecture, quantization and diffusionPipeline. So when another entry with the same format and framework sets autoSelect: true, OME can still pick the runtime for a model that only an autoSelect: false entry matches.

Within one runtime, autoSelect: true entries that share an entry name, or that leave it empty, can't set different priorities. The admission webhook rejects such a runtime with different priorities assigned for the model format.

Tie-breaks¶

For a model with a modelParameterSize, OME adds up how far each bound of a runtime's modelSizeRange is from the model's size. The smaller distance comes first:

• With both bounds, the distance is the range's width, so the narrowest range comes first.
• A runtime without a modelSizeRange has a distance of 0, the smallest possible.
• Since v1.3, a range with one bound counts the distance to that bound. On v1.2.2, set both bounds.

Model size range matching says how OME reads the sizes.

Runtimes that still tie come in ascending order of their names, so runtime-a comes before runtime-b.

To make a size-specific runtime win over one without a range, or to settle a name tie, give the runtime you want a higher priority.

Worked example¶

The BaseModel llama-3-2-3b-instruct from Model version matching lives in llama-demo. It has the format safetensors 1.0.0, the framework transformers 4.45.0.dev0, the architecture LlamaForCausalLM and modelParameterSize: "3.21B". The cluster has four runtimes, each with one entry that matches the model and sets autoSelect: true:

OME ranks them in this order:

1. team-llama, the only ServingRuntime, even though it has the lowest score.
2. vllm-llama. The three ClusterServingRuntimes tie at 6, and it has no size range, so its distance is 0.
3. srt-llama-small, whose range is narrower than srt-llama-large's.
4. srt-llama-large.

To make srt-llama-small the first ClusterServingRuntime, raise its priority to 4. It then scores (1 + 1) × 4 = 8, above vllm-llama's 6, but team-llama still wins in llama-demo. When a runtime changes says when OME picks again.

Check the ranking¶

Since v1.3, kubectl ome runtime explain lists the runtimes in the order OME ranks them. Pass --isvc instead of --model to also apply an InferenceService's accelerator class and deployment mode; see kubectl ome runtime. In the cluster from the worked example:

kubectl ome runtime explain --model llama-3-2-3b-instruct -n llama-demo

RUNTIME           SCOPE        COMPATIBLE   PRIORITY   WEIGHT   REASON
team-llama        Namespaced   Yes          1          2        -
vllm-llama        Cluster      Yes          2          6        -
srt-llama-small   Cluster      Yes          3          6        -
srt-llama-large   Cluster      Yes          3          6        -

The first Yes row is the runtime OME picks. WEIGHT comes from the runtime's first matching entry, so it can differ from the score; Output fields explains each column.

OME also shows the runtime it picked in these places:

The InferenceService's status doesn't record the choice.

Related pages¶

• Serving runtimes: how OME uses the runtime it picks, and when it picks again.
• Troubleshoot runtime selection: find out why OME can't pick a runtime for a model.
• Reference a runtime explicitly: name the runtime instead of letting OME pick one.
• Model size range matching: how OME parses a runtime's modelSizeRange and compares it with the model's size.
• Model version matching: how an entry's versions and operators match the model's.
• kubectl ome runtime: list the runtimes that match a model, in the order that OME ranks them.
• OME API: the fields of SupportedModelFormat.