Run the model agent

The model agent puts the weights of BaseModels and ClusterBaseModels on your nodes. It runs as the DaemonSet ome-model-agent-daemonset. On each node, it gets the files of the models that select the node, reads what each model is, and labels the node when the model is Ready there. OME runs a model's serving pods only on nodes with that label.

Since v1.3, the agent is optional and off by default. Turn it on to serve base models stored on the nodes: hf://, oci://, modelpack://, local:// and vendor:// models. Until you turn it on, those models stay In_Transit, and OME can't serve them. You don't need it for pvc:// models, or for InferenceServices that name only a runtime that loads its own weights.

Step 1: Turn on the agent¶

On v1.2.2, the chart always installs the agent, so go to Step 2. Otherwise, add the value to the values file you install ome-resources with:

modelAgent:
  enabled: true

The agent runs on all nodes whose taints it tolerates, GPU nodes included. Each agent pod requests 10 CPUs and 100Gi of memory, and its priority class, system-node-critical, lets Kubernetes evict lower-priority pods to make room. The agent keeps the weights in the node's /mnt/data/models, which needs room for the models that select the node. To use fewer nodes or another directory, also set gpuNodesOnly, nodeSelector or hostPath from Helm values.

The value needs chart version 1.3.0 or later. Upgrade the release with the chart version you already run, 1.3.0 in this example:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Until v1.3 is released, upgrade from a checkout of main instead, as in Install from source.

Helm prints Release "ome" has been upgraded. Happy Helming!. Wait for the agent pods to start:

kubectl rollout status daemonset/ome-model-agent-daemonset -n ome

While the pods start, kubectl prints Waiting for daemon set lines. When all of them are available, it prints:

daemon set "ome-model-agent-daemonset" successfully rolled out

Step 2: Check the agent on each node¶

Each node that should hold models needs a ready agent pod. List the agent pods with their nodes:

kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
  -o 'custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,READY:.status.containerStatuses[0].ready'

NAME                              NODE         READY
ome-model-agent-daemonset-4xkqz   gpu-node-1   true
ome-model-agent-daemonset-9mfpt   gpu-node-2   true
ome-model-agent-daemonset-tq7wd   gpu-node-3   true

If a node is missing, or a pod shows false, see Troubleshooting.

Step 3: Check where a model is ready¶

Show the label of qwen2-5-7b-instruct on all nodes:

kubectl get nodes \
  -o 'custom-columns=NODE:.metadata.name,STATE:.metadata.labels.models\.ome\.io/clusterbasemodel\.qwen2-5-7b-instruct'

NODE         STATE
gpu-node-1   Ready
gpu-node-2   Ready
gpu-node-3   Updating

The model is ready on gpu-node-1 and gpu-node-2. On gpu-node-3, the agent is still downloading or checking the files, and the label turns Ready, or Failed, when it's done. Nodes outside the model's selector show <none>. For a BaseModel, the label key is models.ome.io/{namespace}.basemodel.{name}. Long names are shortened with a hash, as Node labels and status describes.

The agent also records each model in a ConfigMap named after the node, in ome, under the key clusterbasemodel.{name} or {namespace}.basemodel.{name}. In the JSON entry, status matches the label, config holds the model's details once it's ready, and progress shows how far a Hugging Face download has got. Show the name and state from the entry on gpu-node-1:

kubectl get configmap gpu-node-1 -n ome \
  -o jsonpath='{.data.clusterbasemodel\.qwen2-5-7b-instruct}' | jq '{name, status}'

{
  "name": "qwen2-5-7b-instruct",
  "status": "Ready"
}

Configure the agent¶

Set the agent's values under modelAgent in your values file, and upgrade the release as in Step 1.

Helm values¶

Environment variables¶

Set the agent's environment variables in modelAgent.env, a map of names to values:

modelAgent:
  enabled: true
  env:
    LOG_LEVEL: debug

The chart writes these values into the DaemonSet in plain text, so give each model its token through its Secret instead: see Credentials. Other variables named after the agent's command-line flags have no effect, such as DOWNLOAD_RETRY, which the chart's values.yaml suggests.

How downloads run¶

The chart runs two download workers, so at most two models download at once on a node, and the others wait their turn. After a restart, the agent checks its models again, and downloads go first: the agent re-checks an oci:// model that was Ready before the restart, and that still has its files on the node, only when no download is waiting. Checks of hf:// models, and of models that weren't Ready or that lost their files, queue up with the downloads, so a new model's download can wait behind them.

The agent doesn't retry a failed model by itself. It tries again when its pod restarts, or when a change to the model makes the nodes process it again.

When two oci:// models name the same files, one downloads them, and the other reuses them once they're ready, or downloads them itself after 30 minutes.

With storage.downloadPolicy: ReuseIfExists, models that use the same Hugging Face files share one copy on each node: see Share Hugging Face artifacts.

How a delete stops a download¶

When you delete a model, the agent stops the model's work on each node, so a node doesn't spend hours downloading weights you deleted:

• A download that's waiting its turn never starts: the delete drops it from the queue.
• The agent cancels the model's running download as soon as it sees the deletion. A Hugging Face download stops right away. Since v1.3, an oci:// download stops too: the agent starts no new files, lets a file under 200 MiB finish, and stops the multipart transfer of a larger file mid-file, removing its temporary files. A request to object storage that has already started keeps its retries and returns first, so the stop can wait for one request. On v1.2.2, the agent can't stop an oci:// download once it has started: it runs to completion.

A canceled download doesn't mark the model Failed on the node. The agent then deletes the files that reached the node and removes the model's label, following the rules in Delete a model, and the model's deletion finishes once every node reports its copy deleted.

The .hf-artifact-locks directories¶

The agent keeps lock files in directories named .hf-artifact-locks: one appears next to the path of each hf://, oci:// and modelpack:// model on the node, whatever the model's downloadPolicy. With the paths directly under modelAgent.hostPath, that's one directory for all models, /mnt/data/models/.hf-artifact-locks. Sharing keeps one next to _artifacts too.

A download or deletion holds the model's lock file while it works on the files, and a task that finds the lock held waits and tries again. The locks are file locks on the node's disk, so they hold across agent pods: while a replaced agent pod is still finishing a download, the new pod waits instead of writing to the same path.

The lock files are empty, and hold no lock once their operation ends. Deleting a model leaves them and their directory in place. Don't create, change or delete anything in these directories: removing a lock file in use can let two operations write to or delete the same model's files at once.

TensorRT-LLM models¶

For an oci:// model with modelFormat.name: tensorrtllm, the agent downloads only the objects built for its node's GPU, whose names contain /{gpu}/, such as /H100/. It finds the GPU name by looking up the node's node.kubernetes.io/instance-type label in the ConfigMap model-agent-config-map, and uses an unlisted instance type as it is. When no object matches, the model fails on the node.

Since v1.3, modelAgent.instanceTypeMap doesn't change the ConfigMap, and the map lacks BM.GPU.B300.8, so a model with objects under /B300/ fails on B300 nodes. This is a known bug. Edit the ConfigMap's instance-type-map key instead, and run kubectl rollout restart daemonset/ome-model-agent-daemonset -n ome. After each helm upgrade, check the key, and do both again if your entry is gone. On v1.2.2, add the instance type to modelAgent.instanceTypeMap.

ModelPack models¶

A modelpack:// model is a CNCF ModelPack artifact in a container registry. The agent doesn't talk to the registry itself. It asks a running llmman serve daemon to pull the artifact, runs llmman resolve --no-pull to find where the daemon put the files, and copies them to the model's path, with hard links when the daemon's store is on the same filesystem, so the files cost their bytes once. From there, the model behaves like an hf:// or oci:// model: the agent reads its files, as What OME learns from the model describes, labels the node, and deletes the files when you delete the model.

This ClusterBaseModel pulls the artifact ghcr.io/myorg/llama-3-70b:v1:

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: llama-3-70b
spec:
  storage:
    storageUri: modelpack://ghcr.io/myorg/llama-3-70b:v1
    path: /mnt/data/models/myorg/llama-3-70b

OME ships neither the daemon nor the binary, so provide both to the agent on every node that the model selects:

• A running llmman serve daemon. The agent finds it through LLMMAN_HOST, http://127.0.0.1:17434 by default.
• The llmman binary, on the agent's PATH or at the path in OME_LLMMAN_BIN.

Set both variables in modelAgent.env: see Environment variables. The agent copies the files from the path that llmman resolve reports, and fails when that path doesn't exist in its pod, so mount the binary and the daemon's store into the agent pod with extraVolumes and extraVolumeMounts.

The daemon does the registry work, authentication included, so the model's storage.key and storage.parameters play no role: give the daemon the registry's credentials in its own configuration. When the daemon isn't reachable, the binary is missing, or the pull fails, the model fails on the node: see Troubleshooting.

Metrics¶

The agent serves Prometheus metrics at /metrics on port 8080. Most have the labels model_type, namespace and name. model_type is BaseModel or ClusterBaseModel, and namespace is empty for a ClusterBaseModel:

Since v1.3, podMonitor.enabled: true creates the PodMonitor ome-model-agent for the Prometheus Operator, which needs the Operator's monitoring.coreos.com/v1 CRDs. podMonitor.interval, podMonitor.scrapeTimeout and podMonitor.additionalLabels tune it. Collect metrics covers the other ways to scrape the agent.

Troubleshooting¶

The agent logs what it does with each model. Find the agent pod on a node, here gpu-node-3:

kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
  --field-selector spec.nodeName=gpu-node-3 -o name

pod/ome-model-agent-daemonset-tq7wd

Then read its log lines about the model:

kubectl logs -n ome pod/ome-model-agent-daemonset-tq7wd | grep qwen2-5-7b-instruct

Look for the line that ends in status to Updating before download, where the agent starts on the model, and for errors after it.

The model stays In_Transit or Failed¶

A model stays In_Transit while no node has it Ready or Failed:

• The agent is off: turn it on as in Step 1.
• The model's nodes lack an agent pod: see A node never gets the label.
• The nodes are still downloading it: see A model stays Updating.

When the model is Failed, read the log of the agent on a node where its label is Failed:

After you fix the cause, restart the agent pod on the node, as in A model stays Updating. For an oci:// model, the new agent downloads only the mismatched files.

A model stays Updating¶

A model's label can stay Updating because:

• The agent restarted: see the warning in Configure the agent.
• The model is waiting its turn, or still downloading. For a Hugging Face model, progress in the node's ConfigMap shows how far it has got: see Step 3.
• Another oci:// model is downloading the same files. The log has Same-path model lines that end in before starting duplicate download: see How downloads run.
• The agent has no downloader for the model's scheme, s3://, az://, gs:// or github://, and logs unknown storage type. See Where the weights come from.
• The agent can't parse the oci:// URI, and logs Failed to get target directory path for model with the reason. Use the form from the same section.

After you fix the cause, delete the agent pod on the node. Its replacement processes all the models on the node again:

kubectl delete pod -n ome ome-model-agent-daemonset-tq7wd

kubectl confirms the deletion, and the DaemonSet starts a new agent pod on the node.

A node never gets the label¶

When the model's label never appears on a node:

• The node has no agent pod: check the list from Step 2. The agent's tolerations, gpuNodesOnly, nodeSelector, affinity and resources decide where it fits: see Helm values.
• The model doesn't select the node. Compare its storage.nodeSelector and storage.nodeAffinity with the node's labels: see Choose the nodes.
• The node's labels changed after the agent started. The agent doesn't watch its node's labels, so restart its pod, as in A model stays Updating.
• It's a pvc:// model, which needs no label.

The agent pod isn't ready¶

The pod is ready once it can write to its models directory, modelAgent.hostPath. Ask a pod from Step 2 for its readiness through the API server:

kubectl get --raw "/api/v1/namespaces/ome/pods/ome-model-agent-daemonset-tq7wd:8080/proxy/healthz?verbose"

[+]model-agent-health ok
healthz check passed

When the check fails, kubectl reports an error, and the answer ends in reason withheld. Ask the same pod for /healthz/model-agent-health to see the reason.

When the pod keeps restarting, check that modelAgent.health.port is 8080, and check the path of your hf:// and oci:// models. The agent crashes on each node that an oci:// model with no path field selects. It also crashes on all nodes while an hf:// or oci:// model with no path is being deleted. This is a known bug: give every hf:// and oci:// model a path, and see The agent keeps restarting.

The model's status doesn't list its nodes¶

When the model's label is on the nodes but its status.nodesReady stays empty, check that OME runs in the namespace ome. The agent always reports to ome, and no Helm value changes that, so install OME there.

Clean up¶

Since v1.3, you can turn the agent off. First, delete the BaseModels and ClusterBaseModels stored on the nodes, and wait until they're gone. Each model's deletion waits for the agent on its nodes, as Delete a model describes. Then remove enabled: true from the values, and upgrade the release as in Step 1.

Next steps¶

• Share Hugging Face artifacts: share one copy of a Hugging Face model's files between models.
• Keep downloaded model files: keep a model's files on the nodes after you delete it.
• Serve models from node-local storage: serve weights already on your nodes' disks.
• Base models: how a model chooses its nodes, and what OME learns from it.
• Collect metrics: scrape the agent and the rest of OME.