Serve models from a PVC

Every serving pod reads the model from the same PersistentVolumeClaim, so you keep one copy of the weights, and the model agent downloads nothing to the nodes. You point a BaseModel at the claim with a pvc:// URI, and a metadata Job reads the model's details from the claim. Then the pods of an InferenceService mount the claim read-only. The steps serve meta-llama/Llama-3.2-1B-Instruct.

The pvc:// storage URI¶

A pvc:// URI names the claim and the directory on it that holds the model. Its form depends on the kind of model:

A BaseModel uses a claim in its own namespace. A ClusterBaseModel names the claim's namespace before the colon. Either way, the metadata Job runs in the claim's namespace. The InferenceServices that serve the model must run there too, because a pod can only mount a claim in its own namespace.

The sub-path is required. It's the path from the root of the volume to the directory that holds config.json, or model_index.json for a diffusion model. It can be several levels deep, such as models/llama-3-2-1b-instruct. Serving pods mount it read-only at /opt/ml/model, and OME sets MODEL_PATH to that path.

Step 1: Verify the PVC¶

Check the claim's status and access mode:

kubectl get pvc model-storage -n llama-demo \
  -o 'custom-columns=NAME:.metadata.name,STATUS:.status.phase,ACCESS:.status.accessModes[*]'

NAME            STATUS   ACCESS
model-storage   Bound    ReadOnlyMany

STATUS must be Bound, and ACCESS should be ReadOnlyMany or ReadWriteMany.

Step 2: Create the model¶

Create a BaseModel in llama-demo that points at the claim. For a ClusterBaseModel, see Use a ClusterBaseModel.

Leave out modelArchitecture, modelFormat and the other metadata fields. OME fills them in from the model's files in Step 3, and keeps any that you set.

apiVersion: ome.io/v1beta1
kind: BaseModel
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  storage:
    storageUri: "pvc://model-storage/llama-3-2-1b-instruct"

Apply the file:

kubectl apply -f model.yaml

basemodel.ome.io/llama-3-2-1b-instruct created

Step 3: Watch the model become Ready¶

OME now runs a metadata Job that reads the model's config.json from the claim. From it, OME fills in the model's empty metadata fields and sets the state to Ready. Watch the state:

kubectl get basemodel llama-3-2-1b-instruct -n llama-demo -w \
  -o custom-columns=NAME:.metadata.name,STATE:.status.state

NAME                    STATE
llama-3-2-1b-instruct   In_Transit
llama-3-2-1b-instruct   Ready

Press Ctrl+C once the state is Ready. A state can repeat, and the column shows <none> until OME sets one.

Check what OME recorded:

kubectl get basemodel llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns=ARCHITECTURE:.spec.modelArchitecture,FORMAT:.spec.modelFormat.name,FRAMEWORK:.spec.modelFramework.name,VERSION:.spec.modelFramework.version

ARCHITECTURE       FORMAT        FRAMEWORK      VERSION
LlamaForCausalLM   safetensors   transformers   4.45.0.dev0

VERSION comes from transformers_version in your copy's config.json, so yours can differ. What OME learns from the model lists the other fields that OME records.

Step 4: Deploy an InferenceService¶

Once the model is Ready, create an InferenceService that serves it:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
    kind: BaseModel
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1

spec.model.kind defaults to ClusterBaseModel, so isvc.yaml sets kind: BaseModel. The runtime srt-llama-3-2-1b-instruct declares support for what Step 3 recorded. It sets autoSelect: false, so OME uses it only when an InferenceService names it.

If the runtime doesn't declare support for your copy of the model, OME uses it anyway and records a RuntimeCompatibilityAdvisory event. Since v1.3, kubectl apply also prints a warning. See Reference a runtime explicitly.

Apply the file:

kubectl apply -f isvc.yaml

inferenceservice.ome.io/llama-3-2-1b-instruct created

Wait for the InferenceService to be ready. The first start pulls the SGLang image, which can take several minutes:

kubectl wait --for=condition=Ready inferenceservice/llama-3-2-1b-instruct -n llama-demo --timeout=30m

inferenceservice.ome.io/llama-3-2-1b-instruct condition met

Step 5: Send a request¶

In a second terminal, forward a local port to the engine's Service:

kubectl port-forward -n llama-demo svc/llama-3-2-1b-instruct-engine 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Send a chat completion request. jq prints the model's answer:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Llama-3.2-1B-Instruct", "messages": [{"role": "user", "content": "In one sentence, what is a PersistentVolumeClaim?"}], "max_tokens": 64}' \
  | jq -r '.choices[0].message.content'

A PersistentVolumeClaim (PVC) is a request for storage in Kubernetes that a pod uses to mount a PersistentVolume.

The answer varies from run to run.

Use a ClusterBaseModel¶

A ClusterBaseModel works too, but only InferenceServices in the claim's namespace can use it. Its URI names that namespace:

apiVersion: ome.io/v1beta1
kind: ClusterBaseModel
metadata:
  name: llama-3-2-1b-instruct
spec:
  storage:
    storageUri: "pvc://llama-demo:model-storage/llama-3-2-1b-instruct"

Apply the file in place of the one in Step 2:

kubectl apply -f model.yaml

clusterbasemodel.ome.io/llama-3-2-1b-instruct created

Then continue from Step 3. In the kubectl get basemodel commands, use clusterbasemodel and leave out -n llama-demo. In isvc.yaml, leave out kind: BaseModel. The metadata Job and the InferenceService still run in llama-demo.

Configure the metadata Job¶

The metadata Job's settings are values of the ome-resources chart, under ome.omeAgent.metadataJob. Change them when the claim mounts only on some nodes, or when the Job needs other resources:

A 0 for backoffLimit or ttlSecondsAfterFinished means the default, and an empty resource value sets no request or limit.

The Job runs the ome-agent image, set by ome.omeAgent.image and ome.omeAgent.tag. To pull it from your own registry, see Install from a private registry.

For example, if the claim's CSI driver runs only on the GPU nodes, and those nodes carry the nvidia.com/gpu taint, set nodeSelector and tolerations:

ome:
  omeAgent:
    metadataJob:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule

Add the values to the values file you install ome-resources with. helm list -n ome shows the installed chart version in its CHART column, such as ome-resources-1.2.2. Upgrade the release with that version, so that OME stays in step with the ome-crd chart:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.2.2 -f values.yaml

Pulled: ghcr.io/moirai-internal/charts/ome-resources:1.2.2
Digest: sha256:703ff2932198d1356ec6977e7bba0bdfa013f50088eb96d097d6e49bf88914f5
Release "ome" has been upgraded. Happy Helming!
NAME: ome
LAST DEPLOYED: Mon Sep 28 02:41:07 2026
NAMESPACE: ome
STATUS: deployed
REVISION: 2
TEST SUITE: None

The controller reads these values when it starts. Since v1.3, helm upgrade restarts it when they change. On v1.2.2, restart it yourself:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

New settings apply to the Jobs that OME creates after the restart. To rerun a model's Job with them, delete the Job, as in The metadata Job fails.

Troubleshooting¶

Start with the model's state, and the reason and message on its Ready condition:

kubectl get basemodel llama-3-2-1b-instruct -n llama-demo \
  -o 'custom-columns=STATE:.status.state,REASON:.status.conditions[?(@.type=="Ready")].reason,MESSAGE:.status.conditions[?(@.type=="Ready")].message'

For a missing claim:

STATE    REASON        MESSAGE
Failed   PVCNotFound   PVC llama-demo/model-storage not found

A Ready model shows In_Transit for a moment each time Kubernetes deletes its finished Job after ttlSecondsAfterFinished, because OME runs the Job again. Serving isn't affected.

The metadata Job fails¶

List the model's Jobs and pods with kubectl get jobs,pods -n llama-demo -l models.ome/model-name=llama-3-2-1b-instruct, and read the pods' logs with kubectl logs -n llama-demo -l models.ome/model-name=llama-3-2-1b-instruct:

• If a pod stays Pending, kubectl describe pod shows why. If the claim mounts only on some nodes, set nodeSelector and tolerations as in Configure the metadata Job. A sub-path that's missing from the volume also keeps the pod from starting.
• no model_index.json or config.json found in /model means that the sub-path is wrong, or that user 65532 can't read the directory.
• permission denied means that user 65532 can't read the model's files.

When the fix is a new sub-path in the URI, OME starts a new Job on its own. For other fixes, delete the Job, and OME creates a new one:

kubectl delete job -n llama-demo -l models.ome/model-name=llama-3-2-1b-instruct

job.batch "llama-3-2-1b-instruct-metadata-26cec0b4" deleted from llama-demo namespace

The old failure can show until the new Job finishes.

The webhook rejects the model¶

kubectl apply prints the reason after denied the request:. For a ClusterBaseModel whose URI has no namespace:

ClusterBaseModel PVC URI must specify a namespace (format: pvc://{namespace}:{pvc-name}/{sub-path}), got "pvc://model-storage/llama-3-2-1b-instruct"

For a BaseModel whose URI has a namespace:

namespaced BaseModel PVC URI must not specify a namespace; the BaseModel's own namespace is used, got "pvc://llama-demo:model-storage/llama-3-2-1b-instruct"

Other malformed URIs get invalid PVC storage URI, with the problem at the end. Here the sub-path is missing:

invalid PVC storage URI "pvc://model-storage": invalid PVC storage URI format: missing subpath

Fix the URI to match The pvc:// storage URI, and apply model.yaml again.

Sharded distribution is rejectedUnreleased: coming in v1.3¶

The webhook rejects a pvc:// model with distribution: Sharded:

PVC storage URIs are not compatible with distribution=Sharded; use distribution=PerNode (or omit) for pvc:// models

Leave distribution out, or set it to PerNode.

The webhook rejects the InferenceService¶

Since v1.3, the webhook rejects isvc.yaml until OME records the model's format in Step 3:

runtime srt-llama-3-2-1b-instruct does not support model llama-3-2-1b-instruct: invalid model specification: modelFormat.name: model format name is required

Wait for the model to be Ready, as in Step 3, then apply isvc.yaml again.

Since v1.3, the webhook also rejects an InferenceService that uses the ClusterBaseModel from another namespace, here default:

ClusterBaseModel "llama-3-2-1b-instruct" references PVC in namespace "llama-demo", but InferenceService is in namespace "default"; Kubernetes does not allow pods to mount PVCs from another namespace. Either move the InferenceService to namespace "llama-demo" or replicate the PVC into namespace "default".

Create the InferenceService in llama-demo. For other messages, see The webhook rejects the InferenceService.

Serving pods don't start¶

Run kubectl describe pod -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct, and read the Events at the end of each pod's description:

• If there are no pods, check that spec.model.kind in isvc.yaml matches the kind of model you created. See An InferenceService that serves a BaseModel gets no pods.
• If a scheduling event says that no node fits, check that a node has the free GPU, CPU and memory listed in Before you begin.
• A multi-attach error, or another volume event, means that the pod's node can't mount the claim. Usually another node has mounted a ReadWriteOnce claim. Use a ReadOnlyMany or ReadWriteMany claim.
• If the container starts and then exits, read its logs with kubectl logs -n llama-demo -l ome.io/inferenceservice=llama-3-2-1b-instruct. A sub-path that points at a parent of the model's directory can pass Step 3, but the runtime can't load the model from it. Point the sub-path at the directory that holds config.json.

Clean up¶

Delete the InferenceService and the model:

kubectl delete inferenceservice llama-3-2-1b-instruct -n llama-demo
kubectl delete basemodel llama-3-2-1b-instruct -n llama-demo

inferenceservice.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace
basemodel.ome.io "llama-3-2-1b-instruct" deleted from llama-demo namespace

For a ClusterBaseModel, run kubectl delete clusterbasemodel llama-3-2-1b-instruct instead of the second command.

Deleting the model deletes its metadata Jobs. The claim and the weights on it stay as they are.

OME keeps the ServiceAccount ome-model-metadata in llama-demo and the RoleBinding ome-model-metadata-llama-demo in ome, so that other models in llama-demo can use them. To remove them:

kubectl delete rolebinding -n ome -l models.ome/metadata-source-namespace=llama-demo
kubectl delete serviceaccount ome-model-metadata -n llama-demo

rolebinding.rbac.authorization.k8s.io "ome-model-metadata-llama-demo" deleted from ome namespace
serviceaccount "ome-model-metadata" deleted from llama-demo namespace

OME creates them again the next time it runs a metadata Job in llama-demo.

Next steps¶

• Base models: every storage scheme, and what OME learns from a model.
• Serve models from node-local storage: serve weights that are already on the nodes' disks, with a local:// URI.
• Expose a service without ingress: reach the model through a LoadBalancer or NodePort Service.
• Configure ingress: expose the model through Kubernetes Ingress or Gateway API.