Model version matching

Each entry in a serving runtime's supportedModelFormats can require versions of the model's format and framework, such as safetensors 1.0.0 and transformers 4.45.0.dev0. The entry matches a model only when the format and framework names are equal and the entry's operators accept the model's versions. Watch for three rules:

• A version that only one side sets never matches. When neither side sets one, the versions match.
• GreaterThan and GreaterThanOrEqual set an upper limit on the model's version, not a lower one.
• A version with a suffix, such as the .dev0 in 4.45.0.dev0, matches only an equal version, whatever the operator.

The entry must also match the model's architecture, quantization and diffusion pipeline, and the runtime must pass the other checks that Troubleshoot runtime selection lists.

Where versions and operators are declared¶

A model declares its versions, and a runtime entry declares the versions it accepts, with an operator for each:

Runtime selection ignores the entry's top-level name and version, and the model's operator fields and spec.version. The API reference lists every field of an entry under SupportedModelFormat.

This BaseModel has the format safetensors 1.0.0 and the framework transformers 4.45.0.dev0:

apiVersion: ome.io/v1beta1
kind: BaseModel
metadata:
  name: llama-3-2-3b-instruct
  namespace: llama-demo
  annotations:
    ome.oracle.com/skip-config-parsing: "true"
spec:
  modelType: llama
  modelArchitecture: LlamaForCausalLM
  modelFormat:
    name: safetensors
    version: "1.0.0"
  modelFramework:
    name: transformers
    version: "4.45.0.dev0"
  modelParameterSize: "3.21B"
  modelCapabilities:
    - TEXT_TO_TEXT
  storage:
    storageUri: hf://meta-llama/Llama-3.2-3B-Instruct
    path: /mnt/data/models/meta/llama-3-2-3b-instruct
    key: hf-token

The catalog runtime srt-llama-3-2-3b-instruct accepts them with this entry:

spec:
  supportedModelFormats:
    - modelFramework:
        name: transformers
        version: "4.45.0.dev0"
      modelFormat:
        name: safetensors
        version: "1.0.0"
      modelArchitecture: LlamaForCausalLM
      autoSelect: false
      priority: 1

The entry sets no operator, so both versions must equal the model's. It also sets autoSelect: false, so OME uses this runtime only for an InferenceService that names it.

When OME reads a model's files, it fills in modelFormat and modelFramework, and the annotation on the BaseModel above turns that off. It keeps a modelFramework that you set, even one without a version, but adds the format version it read to a modelFormat that has none. A model whose config.json has no transformers_version gets no framework version.

Format and framework names are case-sensitive: SafeTensors doesn't match safetensors. When the names are equal, the versions decide:

The API requires a modelFramework on every entry, so a model needs one to match any entry.

Operators¶

The entry's operator compares the entry's version, on the left, with the model's:

So the entry's version is the newest one that GreaterThanOrEqual accepts. When either version has a suffix, the ordering operators match only an equal version. Otherwise, they need both versions to have the same number of parts, and the v on both or neither.

This entry accepts transformers versions up to 4.46.0 that have three parts, no v and no suffix, such as 4.44.2. It refuses the BaseModel above, whose 4.45.0.dev0 has a suffix:

spec:
  supportedModelFormats:
    - modelFramework:
        name: transformers
        version: "4.46.0"
        operator: GreaterThanOrEqual
      modelFormat:
        name: safetensors
        version: "1.0.0"
      modelArchitecture: LlamaForCausalLM

Accepted version formats¶

A version has one to three numbers separated by dots, with an optional lowercase v in front. Suffixes can follow the third number:

Nothing checks a version when you apply a model or a runtime. A version that doesn't parse never matches, and version: "" still counts as set, so leave the field out instead.

Examples¶

Mismatch reasons¶

When no entry matches, the runtime's reason is model format '<label>' not in supported formats: followed by each entry's differences, separated by ; . Troubleshoot runtime selection explains the label. These are the differences in names and versions, with example values:

If the BaseModel above had the framework version 4.46.0, srt-llama-3-2-3b-instruct would give this reason:

model format 'mt:safetensors:1.0.0:LlamaForCausalLM:transformers:4.46.0' not in supported formats: framework version mismatch (model=4.46.0, runtime=4.45.0.dev0)

Where a mismatch shows up¶

The webhook rejects only an InferenceService that has an engine section and no runner image of its own, in engine.runner.image or in both engine.leader.runner.image and engine.worker.runner.image.

To fix a mismatch, change the entry's version or operator. If OME read no version from the model's files, set it yourself. Or name the runtime in spec.runtime.name.

Related pages¶

• Troubleshoot runtime selection: find out why OME can't choose a runtime for a model.
• Reference a runtime explicitly: name the runtime instead of letting OME pick one.
• kubectl ome runtime: see which runtimes match a model, and why.
• Serving runtimes: how a runtime declares the models it supports.
• Diffusion pipeline runtime matching: how an entry matches a diffusion model's pipeline.
• Runtime selection scoring: how OME ranks the runtimes that match.
• Runtime accelerator-class matching, Runtime deployment-mode matching and Model size range matching: other checks that can exclude a runtime.