Base models

A BaseModel or ClusterBaseModel lets OME manage a model's weights. You say where the weights live and which nodes keep a copy. OME downloads the weights to those nodes, and runs serving pods only on nodes that hold them. A model on a PersistentVolumeClaim is mounted into the pods instead. OME also reads the model's files to learn its architecture, size and capabilities, and uses them to pick a serving runtime. An InferenceService then serves the model by naming it.

Since v1.3, a base model is optional: an InferenceService can name its runtime and load the weights another way. The model agent that downloads base models is optional too, so turn it on when you install OME.

BaseModel and ClusterBaseModel¶

The two kinds have the same spec and differ in scope:

An InferenceService names its model in spec.model.name, and the model's kind in spec.model.kind, which defaults to ClusterBaseModel. To serve a BaseModel, set kind: BaseModel. With the wrong kind, OME admits the InferenceService but doesn't create or update its serving workload; see An InferenceService that serves a BaseModel gets no pods.

Give BaseModels and ClusterBaseModels different names. When both kinds share a name, InferenceServices in the BaseModel's namespace get the BaseModel, even when they ask for the ClusterBaseModel.

This ClusterBaseModel downloads Qwen/Qwen2.5-7B-Instruct from Hugging Face to /mnt/data/models/qwen/qwen2-5-7b-instruct on the nodes that run the model agent:

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: qwen2-5-7b-instruct
spec:
  storage:
    storageUri: hf://Qwen/Qwen2.5-7B-Instruct
    path: /mnt/data/models/qwen/qwen2-5-7b-instruct

Apply the file:

kubectl apply -f model.yaml

clusterbasemodel.ome.io/qwen2-5-7b-instruct created

Set storage.storageUri, and for hf://, oci://, modelpack:// and local:// models, storage.path. OME fills in the rest of the metadata from the model's files, and the OME API reference lists every field.

Where the weights come from¶

storage.storageUri says where the weights are, and its scheme decides what OME does:

Credentials¶

hf:// models read their token from a Secret that you name in storage.key. The agent reads the Secret from the model's namespace, or from ome for a ClusterBaseModel. The token comes from the Secret's token key, or from the key in storage.parameters.secretKey.

Create the namespace llama-demo, and a Secret in it with your Hugging Face token:

kubectl create namespace llama-demo
kubectl create secret generic hf-token -n llama-demo --from-literal=token="$HF_TOKEN"

namespace/llama-demo created
secret/hf-token created

This BaseModel downloads the gated model meta-llama/Llama-3.1-8B-Instruct with the token:

apiVersion: ome.io/v1beta1
kind: BaseModel
metadata:
  name: llama-3-1-8b-instruct
  namespace: llama-demo
spec:
  storage:
    storageUri: hf://meta-llama/Llama-3.1-8B-Instruct
    path: /mnt/data/models/meta/llama-3-1-8b-instruct
    key: hf-token

The agent uses the first token it finds:

1. The Secret in storage.key.
2. storage.parameters.token, which leaves the token in plain text in the model's spec, so prefer a Secret.
3. The agent's own HF_TOKEN environment variable, if you set one.

Without a token, a gated model fails to download, and the node reports Failed.

oci:// models authenticate with the method in storage.parameters.auth: InstancePrincipal, the default, UserPrincipal, ResourcePrincipal or OkeWorkloadIdentity. storage.parameters.region overrides the region.

modelpack:// models take no credentials from the model: the llmman serve daemon talks to the registry, so storage.key and storage.parameters play no role. Give the daemon the registry's credentials in its own configuration. See ModelPack models.

How weights reach the nodes¶

The model agent puts the weights on the nodes. It runs as the DaemonSet ome-model-agent-daemonset, by default on all nodes whose taints it tolerates, including nodes with the nvidia.com/gpu taint. Helm values lists the settings that choose its nodes.

Since v1.3, the ome-resources chart installs the agent only when you set modelAgent.enabled: true. On v1.2.2, the chart always installs it. Without the agent, models stay In_Transit, except pvc:// models.

storage.path is the directory on the nodes that holds the model. Serving pods mount it read-only at the same path, and OME sets the MODEL_PATH environment variable to it, which the pre-configured runtimes load the model from.

The agent sees the host directory in modelAgent.hostPath, /mnt/data/models by default. Keep path under it, or mount more directories with modelAgent.extraVolumes and modelAgent.extraVolumeMounts.

With storage.downloadPolicy: ReuseIfExists, models that use the same Hugging Face files share one copy on each node. The default, AlwaysDownload, gives each model its own copy. See Share Hugging Face artifacts.

Choose the nodes¶

By default, all nodes that run the agent keep a copy of the model. Two storage fields narrow that down, and a node must match both when you set both:

This BaseModel keeps a copy on the nodes of two instance types:

apiVersion: ome.io/v1beta1
kind: BaseModel
metadata:
  name: llama-3-3-70b-instruct
  namespace: llama-demo
spec:
  storage:
    storageUri: hf://meta-llama/Llama-3.3-70B-Instruct
    path: /mnt/data/models/meta/llama-3-3-70b-instruct
    key: hf-token
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                  - BM.GPU.H100.8
                  - BM.GPU.H200.8

When you change the model or a node:

While a node processes the model, the model's label there is Updating, so new serving pods avoid that node until it's Ready again. Running pods keep running.

Node labels and status¶

The agent labels each selected Node with the model's state there. The label key depends on the kind:

When a ClusterBaseModel's name is longer than 32 characters, or a BaseModel's namespace and name together are longer than 38, OME shortens the key with a hash. Read the key from the Node's labels then.

The label is Updating while the agent downloads or checks the model, then Ready or Failed.

The model's status lists the nodes where it's ready and where it failed:

kubectl get clusterbasemodel qwen2-5-7b-instruct \
  -o 'custom-columns=STATE:.status.state,READY:.status.nodesReady[*],FAILED:.status.nodesFailed[*]'

STATE   READY                   FAILED
Ready   gpu-node-1,gpu-node-2   <none>

Or select the nodes by label:

kubectl get nodes -l models.ome.io/clusterbasemodel.qwen2-5-7b-instruct=Ready -o name

node/gpu-node-1
node/gpu-node-2

OME runs an InferenceService's serving pods only on nodes where the model's label is Ready. pvc:// models have no copy on the nodes, so their pods have no such constraint.

Model lifecycle¶

status.state is the model's state across the cluster, shown in the READY column of kubectl get basemodel and kubectl get clusterbasemodel:

A model is Ready as soon as one node has it, so some serving pods can stay Pending while other nodes download it. If a model stays In_Transit or becomes Failed, see Troubleshooting in the model agent guide, or in the PVC guide for a pvc:// model.

Create an InferenceService once its model is Ready. OME needs the model's format, which it reads from the model's files, to choose or check a runtime. Without it, the admission webhook rejects the InferenceService with a message that ends in model format name is required.

Disable a model¶

To stop new use of a model, set disabled: true in its spec. The admission webhook then rejects new and changed InferenceServices that use it, with model qwen2-5-7b-instruct is disabled. OME also stops creating or updating their serving workloads, and records a ModelReconcileError event with specified base model qwen2-5-7b-instruct is disabled. Running pods keep running, and the weights stay on the nodes.

Delete a model¶

When you delete a model, a finalizer keeps it until the agent on each node that recorded the model reports its copy deleted. If a node's agent isn't running, the deletion waits until it runs again or the node leaves the cluster.

You don't have to wait for the download. On a node that's still downloading the model, the agent cancels the download, and a queued download never starts, so the nodes don't keep downloading weights you deleted. A canceled download doesn't mark the model Failed on the node. How a delete stops a download in the model agent guide says how quickly a running download stops; on v1.2.2, the agent can't stop an oci:// download that has started.

On each node, the agent removes the model's label and deletes the files of an hf://, oci:// or modelpack:// model from path, keeping every file that another model, in any namespace, still uses. A model with the same path — even written with a trailing slash, or reached through a symlink — keeps the whole directory, and a model whose path leads inside it keeps its own files and their parent directories while the rest goes; When the agent deletes files in the retention guide has the exact rules. The agent also keeps the whole directory when the model has the label models.ome/reserve-model-artifact: "true". It leaves local:// and vendor:// files in place. Files shared through ReuseIfExists follow the rules in Share Hugging Face artifacts. For pvc:// models, see Clean up in the PVC guide.

Set the reserve label when you create the model: adding a label later makes all matching nodes process the model again. See Keep downloaded model files.

What OME learns from the model¶

Once a node has the weights, the agent reads the model's config.json, or model_index.json for a diffusion pipeline, and hf_quant_config.json if there is one. OME fills in these fields:

OME fills in only empty fields, so a value that you set always wins. For pvc:// models, a metadata Job fills in the same fields, except modelConfiguration and diffusionPipeline.

Set the metadata yourself¶

Set these fields yourself:

• modelFormat and modelFramework, for vendor:// models and models with unreadable files, such as a directory without config.json. These models still become Ready, but OME needs both fields to choose a runtime.
• modelCapabilities, when OME doesn't recognize the architecture and leaves an empty entry.
• displayName, version, vendor and apiCapabilities, which OME never fills in.

To skip reading a model's files, add the annotation ome.oracle.com/skip-config-parsing: "true". OME then keeps only the metadata you set. For a pvc:// model, the metadata Job ignores the annotation and fills in the fields you left empty. This BaseModel sets the metadata that the runtime srt-llama-3-2-3b-instruct supports:

apiVersion: ome.io/v1beta1
kind: BaseModel
metadata:
  name: llama-3-2-3b-instruct
  namespace: llama-demo
  annotations:
    ome.oracle.com/skip-config-parsing: "true"
spec:
  modelType: llama
  modelArchitecture: LlamaForCausalLM
  modelFormat:
    name: safetensors
    version: "1.0.0"
  modelFramework:
    name: transformers
    version: "4.45.0.dev0"
  modelParameterSize: "3.21B"
  modelCapabilities:
    - TEXT_TO_TEXT
  storage:
    storageUri: hf://meta-llama/Llama-3.2-3B-Instruct
    path: /mnt/data/models/meta/llama-3-2-3b-instruct
    key: hf-token

How runtimes match the model¶

A serving runtime lists the models it supports in supportedModelFormats, by format, framework, architecture and quantization, and can limit their size with modelSizeRange. Model version matching and Diffusion pipeline runtime matching explain when a model matches.

When an InferenceService doesn't name a runtime, OME chooses among the matching runtimes that have an entry with autoSelect: true. A ServingRuntime in the InferenceService's namespace wins over any ClusterServingRuntime, and among runtimes of the same kind, the highest score wins.

When an InferenceService names a runtime, OME uses it even if the runtime doesn't declare support for the model, and records a RuntimeCompatibilityAdvisory event. Sharded models are the exception; see Distribution.

DistributionUnreleased: coming in v1.3¶

spec.distribution decides how the weights are laid out. PerNode, the default, puts a full copy on each selected node; leave the field out to get it.

Sharded is for loading the model in chunks from a model cache provider, which this release can't configure yet. The admission webhook rejects InferenceServices that use a Sharded model, even when they name a runtime. The message includes no model cache provider is configured for sharded model loading.

Fine-tuned weights¶

A FineTunedWeight holds weights fine-tuned from the base model in its baseModelRef. To serve it, name the base model in spec.model.name and the FineTunedWeight in spec.model.fineTunedWeights. OME serves one fine-tuned weight per InferenceService.

Next steps¶

• Serve models from a PVC: serve weights from a PersistentVolumeClaim.
• Serve models from node-local storage: serve weights already on the nodes' disks.
• Run the model agent: install, tune and troubleshoot the agent.
• Share Hugging Face artifacts: keep one copy of shared Hugging Face files per node.
• Serving runtimes: how a runtime declares the models it supports.