Keep downloaded model files

When you delete a base model, or change the nodes it selects, the model agent deletes the files it downloaded there. Give the model the label models.ome/reserve-model-artifact: "true" first, and the files stay. You can then create the model again later, even after you reinstall OME, and serve it without downloading the weights again.

Step 1: Add the reserve label¶

The steps use the ClusterBaseModel qwen2-5-7b-instruct, from hf://Qwen/Qwen2.5-7B-Instruct, on three GPU nodes: gpu-node-1, gpu-node-2 and gpu-node-3. Its path is under /mnt/data/models, the default modelAgent.hostPath.

The label works on BaseModels and ClusterBaseModels, and its value is true in any letter case. Its key starts with models.ome/, not models.ome.io/ like the node labels. Add it before you delete the model or change its nodeSelector or nodeAffinity: a label that you add afterwards doesn't bring the files back.

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: qwen2-5-7b-instruct
  labels:
    models.ome/reserve-model-artifact: "true"
spec:
  storage:
    storageUri: hf://Qwen/Qwen2.5-7B-Instruct
    path: /mnt/data/models/qwen/qwen2-5-7b-instruct

kubectl apply -f model.yaml

clusterbasemodel.ome.io/qwen2-5-7b-instruct created

kubectl label clusterbasemodel qwen2-5-7b-instruct models.ome/reserve-model-artifact=true

clusterbasemodel.ome.io/qwen2-5-7b-instruct labeled

Check that the model is Ready on the nodes:

kubectl get nodes \
  -o 'custom-columns=NODE:.metadata.name,STATE:.metadata.labels.models\.ome\.io/clusterbasemodel\.qwen2-5-7b-instruct'

NODE         STATE
gpu-node-1   Ready
gpu-node-2   Ready
gpu-node-3   Ready

Step 2: Delete the model and check the files¶

Delete the model:

kubectl delete clusterbasemodel qwen2-5-7b-instruct

clusterbasemodel.ome.io "qwen2-5-7b-instruct" deleted

kubectl then waits until the agents have processed the deletion and OME has removed the model. The model's label is gone from the nodes:

kubectl get nodes \
  -o 'custom-columns=NODE:.metadata.name,STATE:.metadata.labels.models\.ome\.io/clusterbasemodel\.qwen2-5-7b-instruct'

NODE         STATE
gpu-node-1   <none>
gpu-node-2   <none>
gpu-node-3   <none>

The files stay on the nodes. To see them, find the agent pod on a node, here gpu-node-1:

kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
  --field-selector spec.nodeName=gpu-node-1 -o name

pod/ome-model-agent-daemonset-4xkqz

The agent pod mounts the models directory at the node's path, so check for a model file through it:

kubectl exec -n ome ome-model-agent-daemonset-4xkqz -- ls /mnt/data/models/qwen/qwen2-5-7b-instruct/config.json

/mnt/data/models/qwen/qwen2-5-7b-instruct/config.json

Reuse the kept files¶

To serve the kept files again, create a model with the same storageUri and path. The agents check the files against the source, and download only the ones that are missing or differ. The model becomes Ready without downloading the kept files again. This applies to the default downloadPolicy, AlwaysDownload. For ReuseIfExists models, see Share Hugging Face artifacts.

To serve the files without downloading anything, create a local:// model with the directory as its URI and its path, such as local:///mnt/data/models/qwen/qwen2-5-7b-instruct. See Serve models from node-local storage.

When the agent deletes files¶

The agent deletes a node's copy of an hf://, oci:// or modelpack:// model when you delete the model. It also deletes it when you change the model's storage.nodeSelector or storage.nodeAffinity and the node stops matching. It deletes the storage.path directory, including files that OME didn't put there, but keeps every file that another model's path still reaches. The source on Hugging Face or in object storage stays. You manage the files of local:// and pvc:// models yourself: for a PVC, see Clean up in the PVC guide.

The agent keeps files in these cases:

When another model's path is /mnt/data/models/qwen/qwen2-5-7b-instruct/releases/rev-002, for example, deleting a model without the reserve label whose path is /mnt/data/models/qwen/qwen2-5-7b-instruct keeps releases/rev-002 and the directories that lead to it, and removes config.json, the weights and every other unreferenced file and directory. Deleting the last model that uses a directory deletes the files, unless it has the reserve label.

When the agent can't list the BaseModels and ClusterBaseModels to find these references, it keeps all the files. The deletion then completes without removing them, or fails, as in The model stays after you delete it. The agent also leaves every .hf-artifact-locks directory and _artifacts directory in place.

When it keeps files, the agent still removes the deleted model's label from the node, and OME stops tracking the files. They stay until you delete them, as in Clean up.

Troubleshooting¶

The files are gone after a deletion¶

Create the model again to download the files, and add the label before the next deletion. The usual causes:

• The label came too late. It must be on the model before you delete it or change its nodes.
• The key or value is wrong. The key is models.ome/reserve-model-artifact, and the value is true.
• Another model's deletion removed them, as When the agent deletes files describes.

The model stays after you delete it¶

The model stays until the agents on its nodes report their copies deleted. The controller logs the nodes it's waiting for:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'Waiting for model to be cleared'

Each line names the model, and lists in nodes the nodes that still have it. On those nodes:

• If the agent isn't running, the deletion resumes when it starts again. If the node is gone for good, delete its Node object, and OME stops waiting for it.
• If the agent couldn't delete the files, its log has Failed to delete model or Failed to delete Hugging Face model, with the error. Fix the cause, such as the directory's permissions, then delete the agent pod, as in A model stays Updating. The new pod retries the deletion when it starts.
• If the agent keeps restarting, see The agent keeps restarting.

The agent keeps restarting¶

When the agent pods on every node restart again and again after you delete an hf:// or oci:// model, check the model's path. The lists in Clean up show <none> for a model that has no path field, and deleting such a model crashes the agent. This is a known bug. Set the model's path to an empty string:

kubectl patch clusterbasemodel qwen2-5-7b-instruct --type merge -p '{"spec":{"storage":{"path":""}}}'

clusterbasemodel.ome.io/qwen2-5-7b-instruct patched

For a BaseModel, run kubectl patch basemodel <name> -n <namespace> with the same patch. The agents then process the deletion, and OME removes the model. Use an empty string, not another directory: the agents could delete that directory.

An oci:// model with no path also crashes the agent when the agent downloads it. Give every hf:// and oci:// model a path, as How weights reach the nodes describes.

Clean up¶

Before you delete the kept files, check that no model uses the directory. List the path of every model:

kubectl get clusterbasemodels -o 'custom-columns=NAME:.metadata.name,PATH:.spec.storage.path'

NAME   PATH

kubectl get basemodels -A -o 'custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PATH:.spec.storage.path'

NAMESPACE   NAME   PATH

The example cluster has no other model, so the lists are empty. In your cluster, no model's path should be /mnt/data/models/qwen/qwen2-5-7b-instruct or a directory inside it. Then delete the directory through the agent pod on each node that kept it, starting with gpu-node-1:

kubectl exec -n ome ome-model-agent-daemonset-4xkqz -- rm -r /mnt/data/models/qwen/qwen2-5-7b-instruct

The command prints nothing when it succeeds. Repeat it with the agent pods on gpu-node-2 and gpu-node-3.

Next steps¶

• Run the model agent: the agent's settings, and how it downloads and checks models.
• Share Hugging Face artifacts: keep one copy of a Hugging Face model's files on each node for several models.
• Base models: how a model chooses its nodes, and what happens when you delete it.
• Serve models from node-local storage: serve files that are already on the nodes.
• Labels and annotations: the labels that OME reads, including models.ome/reserve-model-artifact.