Serve models from node-local storage

Serve a model from weights that are already on your GPU nodes' disks, with nothing to download. You point a ClusterBaseModel at their directory with a local:// URI. OME then runs the pods of an InferenceService only on the nodes that hold the files, with the directory mounted read-only. The model agent checks the files on each node, and OME never deletes them. Since v1.3, a base model is optional, but it's what tells OME where these weights are. The steps serve meta-llama/Llama-3.2-1B-Instruct.

The local:// URI and path¶

A local:// URI is local:// followed by the directory's absolute path, so it has three slashes: local:///mnt/data/models/llama-3-2-1b-instruct.

Set storage.path to the same directory. The agent checks the directory in path, and the serving pods mount it at the same path, with MODEL_PATH set to it.

Step 1: Stage the weights and label the nodes¶

Copy the model's files into the same directory on each node that should serve the model. Use your own tools, such as your node image or a copy job. Put config.json and the weight files at the top of the directory, because the runtime loads them from there. By default, the agent sees only the host directory in modelAgent.hostPath, /mnt/data/models, so the steps use /mnt/data/models/llama-3-2-1b-instruct on gpu-node-1 and gpu-node-2. For another directory, see Use another directory.

Check that the agent on each node sees the files:

for node in gpu-node-1 gpu-node-2; do
  pod=$(kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
    --field-selector spec.nodeName=$node -o name)
  echo "$node: $(kubectl exec -n ome $pod -- ls /mnt/data/models/llama-3-2-1b-instruct/config.json)"
done

gpu-node-1: /mnt/data/models/llama-3-2-1b-instruct/config.json
gpu-node-2: /mnt/data/models/llama-3-2-1b-instruct/config.json

If ls reports No such file or directory on a node, check the path there.

The model in Step 2 selects the two nodes by a label. Without a node selector, the agent on every other node marks the model Failed there. If the two nodes already share a label that other nodes lack, such as their node pool's label, use it in Step 2 and skip the rest of this step. Otherwise, label them:

kubectl label node gpu-node-1 gpu-node-2 weights.example.com/llama-3-2-1b-instruct=true

node/gpu-node-1 labeled
node/gpu-node-2 labeled

The agent reads its node's labels when it starts, so restart the agent pods on the two nodes. The DaemonSet starts new ones:

for node in gpu-node-1 gpu-node-2; do
  kubectl delete pod -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
    --field-selector spec.nodeName=$node
done

pod "ome-model-agent-daemonset-4xkqz" deleted from ome namespace
pod "ome-model-agent-daemonset-9mfpt" deleted from ome namespace

The new agents check all the models on their nodes again, so those models show Updating there for a while.

Step 2: Create the model¶

Create a ClusterBaseModel whose URI and path both name the directory from Step 1, and whose node selector matches the nodes' label:

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: llama-3-2-1b-instruct-local
spec:
  storage:
    storageUri: "local:///mnt/data/models/llama-3-2-1b-instruct"
    path: /mnt/data/models/llama-3-2-1b-instruct
    nodeSelector:
      weights.example.com/llama-3-2-1b-instruct: "true"

Leave out modelArchitecture, modelFormat and the other metadata fields. OME fills in the empty ones from the model's files in Step 3, and keeps any that you set.

Use only lowercase letters, digits and hyphens in the model's name. OME names the serving pods' volume after the model, and Kubernetes rejects a volume name with a dot, such as llama-3.2-1b-instruct.

For a namespaced model, see Use a BaseModel.

Apply the file:

kubectl apply -f model.yaml

clusterbasemodel.ome.io/llama-3-2-1b-instruct-local created

Step 3: Watch the model become Ready¶

On each node that the model selects, the agent checks the directory and reads the model's metadata from config.json. Watch the model's state:

kubectl get clusterbasemodel llama-3-2-1b-instruct-local -w \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state

NAME                          STATE
llama-3-2-1b-instruct-local   In_Transit
llama-3-2-1b-instruct-local   Ready

Press Ctrl+C once the state is Ready. The state can repeat, or show <none> at first. After the restart in Step 1, this can take a while: see How downloads run.

The model is Ready as soon as one node is, so check both nodes:

kubectl get clusterbasemodel llama-3-2-1b-instruct-local \
  -o 'custom-columns=STATE:.status.state,READY:.status.nodesReady[*],FAILED:.status.nodesFailed[*]'

STATE   READY                   FAILED
Ready   gpu-node-1,gpu-node-2   <none>

Both nodes should be in READY. If a node is missing there, or the model stays In_Transit, see The model isn't ready on a node.

Check what OME recorded from config.json:

kubectl get clusterbasemodel llama-3-2-1b-instruct-local \
  -o custom-columns=ARCHITECTURE:.spec.modelArchitecture,FORMAT:.spec.modelFormat.name,FRAMEWORK:.spec.modelFramework.name,VERSION:.spec.modelFramework.version

ARCHITECTURE       FORMAT        FRAMEWORK      VERSION
LlamaForCausalLM   safetensors   transformers   4.45.0.dev0

VERSION comes from transformers_version in your copy's config.json, so yours can differ. If FORMAT is <none>, the agents failed to read config.json. Fix that before Step 4, as in The webhook rejects the InferenceService.

Step 4: Deploy an InferenceService¶

Once the model is Ready, create an InferenceService that serves it:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct-local
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

spec.model.kind defaults to ClusterBaseModel, so isvc.yaml leaves it out. The runtime sets autoSelect: false, so OME uses it only when an InferenceService names it.

If the runtime doesn't declare support for your copy of the model, OME uses it anyway and records a RuntimeCompatibilityAdvisory event. Since v1.3, kubectl apply also prints a warning. See Reference a runtime explicitly.

Apply the file:

kubectl apply -f isvc.yaml

inferenceservice.ome.io/llama-3-2-1b-instruct created

Wait for the InferenceService to be ready. The first start pulls the SGLang image, which can take several minutes:

kubectl wait --for=condition=Ready inferenceservice/llama-3-2-1b-instruct -n llama-demo --timeout=30m

inferenceservice.ome.io/llama-3-2-1b-instruct condition met

Check the pod's node and its model volume:

kubectl get pods -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct \
  -o 'custom-columns=NODE:.spec.nodeName,HOSTPATH:.spec.volumes[?(@.name=="llama-3-2-1b-instruct-local")].hostPath.path,MOUNTPATH:.spec.containers[*].volumeMounts[?(@.name=="llama-3-2-1b-instruct-local")].mountPath,READONLY:.spec.containers[*].volumeMounts[?(@.name=="llama-3-2-1b-instruct-local")].readOnly'

NODE         HOSTPATH                                 MOUNTPATH                                READONLY
gpu-node-1   /mnt/data/models/llama-3-2-1b-instruct   /mnt/data/models/llama-3-2-1b-instruct   true

Step 5: Send a request¶

In a second terminal, forward a local port to the engine's Service:

kubectl port-forward -n llama-demo svc/llama-3-2-1b-instruct-engine 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Send a chat completion request. jq prints the model's answer:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.2-1B-Instruct", "messages": [{"role": "user", "content": "In one sentence, what is a hostPath volume?"}], "max_tokens": 64}' \
  | jq -r '.choices[0].message.content'

A hostPath volume mounts a file or directory from the node's filesystem into a pod.

The answer varies from run to run.

Use another directory¶

To keep the weights outside /mnt/data/models, set modelAgent.hostPath to their directory, as Install OME describes. Or mount their directory into the agent as well, at the same path as on the host, because serving pods mount the host directory that path names. This values file mounts /raid/models:

modelAgent:
  enabled: true
  extraVolumes:
    - name: raid-models
      hostPath:
        path: /raid/models
        type: DirectoryOrCreate
  extraVolumeMounts:
    - name: raid-models
      mountPath: /raid/models

DirectoryOrCreate creates the directory on nodes that lack it, so the agent pod starts there too. Add the values to the values file you install ome-resources with, then upgrade the release, as in Step 1 of Run the model agent.

Use a BaseModel¶

To make the model available only in llama-demo, create a BaseModel there instead, with the same spec. In the commands on the model, use basemodel and -n llama-demo. In isvc.yaml, add kind: BaseModel under model. The model's label on the nodes then has the key models.ome.io/llama-demo.basemodel.llama-3-2-1b-instruct-local.

Troubleshooting¶

The model isn't ready on a node¶

Check the model's label on every node:

kubectl get nodes \
  -o 'custom-columns=NODE:.metadata.name,STATE:.metadata.labels.models\.ome\.io/clusterbasemodel\.llama-3-2-1b-instruct-local'

For example, when the files are missing on gpu-node-2:

NODE         STATE
gpu-node-1   Ready
gpu-node-2   Failed
gpu-node-3   <none>

Read the agent's log on that node, here gpu-node-2:

pod=$(kubectl get pods -n ome -l app.kubernetes.io/component=ome-model-agent-daemonset \
  --field-selector spec.nodeName=gpu-node-2 -o name)
kubectl logs -n ome $pod | grep llama-3-2-1b-instruct

On a Failed node, look for one of these lines:

The agent doesn't retry a failed model on its own. After you fix the files, change an annotation on the model, with a new value each time, and the agents check it again:

kubectl annotate clusterbasemodel llama-3-2-1b-instruct-local example.com/recheck=1 --overwrite

clusterbasemodel.ome.io/llama-3-2-1b-instruct-local annotated

The webhook rejects the InferenceService¶

Since v1.3, the webhook rejects isvc.yaml until OME records the model's format in Step 3:

runtime srt-llama-3-2-1b-instruct does not support model llama-3-2-1b-instruct-local: invalid model specification: modelFormat.name: model format name is required

Wait until the model is Ready, as in Step 3, then apply isvc.yaml again.

If the model is Ready but FORMAT was <none> in Step 3, the agents failed to read config.json. Their logs have Failed to parse and update model config for local model, with the reason. Put config.json at the top of the model's directory on each node, and make the agents check the model again. Or set modelFormat and modelFramework in model.yaml yourself, as in Set the metadata yourself.

The serving pods aren't created¶

If the InferenceService has no pods, look for events about pods that Kubernetes refused to create:

kubectl get events -n llama-demo --field-selector reason=FailedCreate

When Pod Security rejects the hostPath volume, the message contains hostPath volumes at the baseline level, or restricted volume type "hostPath" at the restricted level. Use a namespace at the privileged level, as in Before you begin.

With no such events, read the controller's logs with kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1:

• a lowercase RFC 1123 label must consist of means that the model's name has a dot. Create the model again under a name without dots, and use the new name in isvc.yaml.
• For a BaseModel, check that isvc.yaml sets kind: BaseModel. See An InferenceService that serves a BaseModel gets no pods.

The serving pods don't start¶

Run kubectl describe pod -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct, and read the Events at the end of each pod's description:

• If a scheduling event says that no node fits, check that a node with the model's Ready label also has a free GPU, 10 CPUs and 30 GiB of memory.
• If the container starts and then exits, read its logs with kubectl logs -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct. If they show the literal text $(MODEL_PATH), see The serving pods have no model volume. Files in a subdirectory can pass Step 3, but the runtime loads the model from the top of the directory, so move them up.

The serving pods have no model volume¶

When the pod check in Step 4 shows <none> for the model volume, the model is missing storage.path. Add path to model.yaml, with the same directory as the URI, and apply it. The agents check the model again on their own.

A change to the model doesn't reach the serving pods, so delete the InferenceService and create it again:

kubectl delete inferenceservice llama-3-2-1b-instruct -n llama-demo
kubectl apply -f isvc.yaml

inferenceservice.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace
inferenceservice.ome.io/llama-3-2-1b-instruct created

Clean up¶

Delete the InferenceService and the model:

kubectl delete inferenceservice llama-3-2-1b-instruct -n llama-demo
kubectl delete clusterbasemodel llama-3-2-1b-instruct-local

inferenceservice.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace
clusterbasemodel.ome.io "llama-3-2-1b-instruct-local" deleted

If you labeled the nodes in Step 1, remove the label:

kubectl label node gpu-node-1 gpu-node-2 weights.example.com/llama-3-2-1b-instruct-

node/gpu-node-1 unlabeled
node/gpu-node-2 unlabeled

To free the disk space, delete /mnt/data/models/llama-3-2-1b-instruct on each node yourself.

Next steps¶

• Serve models from a PVC: serve weights from a PersistentVolumeClaim, with no copy on each node.
• Base models: every storage scheme, how OME picks the nodes, and what it learns from a model.
• Run the model agent: the agent's settings, its logs, and how it processes models.