Install OME

Install OME with Helm, and check that the OME controller and the optional model agent are running. First you install cert-manager, which issues the certificate for OME's admission webhooks. Then you install two OME charts: ome-crd adds OME's custom resources, and ome-resources runs the controller and, when you turn it on, the model agent. How OME works explains each component.

The commands install v1.2.2, the latest release. The features marked Since v1.3, such as OMENative, Alfred and the OME scheduler, are in main but not yet in a release. To use them now, install from source.

Step 1: Install cert-manager and the PodMonitor CRD¶

If your cluster already runs cert-manager, skip to The PodMonitor CRD. Otherwise, install it with its CRDs:

helm upgrade --install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.21.2 \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

Helm installs cert-manager and reports the release's status as deployed, followed by cert-manager's notes. The ome-resources chart brings its own issuer, so you don't need to create one.

Wait for cert-manager to be available:

kubectl wait --for=condition=Available deployment --all -n cert-manager --timeout=5m

deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met

The PodMonitor CRDUnreleased: coming in v1.3¶

RawDeployment and MultiNode (deprecated) components need the Prometheus Operator's PodMonitor CRD, or their InferenceServices never become Ready. RawDeployment is the default for a component without a leader and workers, such as the engine in Serve your first model. If you run the Prometheus Operator, you already have the CRD. Otherwise, install the CRD on its own:

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.79.2/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml

customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com serverside-applied

The controller looks for the CRD only when it starts. If OME is already running, restart it with kubectl rollout restart deployment/ome-controller-manager -n ome.

Step 2: Install the CRDs¶

Install the ome-crd chart, which adds OME's CustomResourceDefinitions, as the release ome-crd:

helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd \
  --version 1.2.2 \
  --namespace ome --create-namespace

Helm installs the release and reports its status as deployed.

--version takes the release's version without the leading v. Install the same version of both OME charts, and pass --version on every upgrade, or Helm installs the latest.

Check that the API server serves OME's resources:

kubectl api-resources --api-group=ome.io

NAME                     SHORTNAMES   APIVERSION       NAMESPACED   KIND
acceleratorclasses                    ome.io/v1beta1   false        AcceleratorClass
basemodels                            ome.io/v1beta1   true         BaseModel
benchmarkjobs                         ome.io/v1beta1   true         BenchmarkJob
clusterbasemodels                     ome.io/v1beta1   false        ClusterBaseModel
clusterservingruntimes                ome.io/v1beta1   false        ClusterServingRuntime
finetunedweights                      ome.io/v1beta1   false        FineTunedWeight
inferenceservices        isvc         ome.io/v1beta1   true         InferenceService
servingruntimes                       ome.io/v1beta1   true         ServingRuntime

Since v1.3, the list also has acceleratorquotas, inferencereplicas, trafficmaps and workloadclusters.

Step 3: Install OME¶

The ome-resources chart runs the OME controller, as the Deployment ome-controller-manager, and the model agent, as the DaemonSet ome-model-agent-daemonset. The model agent downloads the weights of BaseModels and ClusterBaseModels to your nodes. Since v1.3, it's optional and off by default. Models on a PersistentVolumeClaim and runtimes that fetch their own weights don't need it.

Keep your settings for the chart in a values file. This one turns on the model agent, because Serve your first model uses a ClusterBaseModel:

modelAgent:
  enabled: true

Install the chart as the release ome, with the values file:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --version 1.2.2 \
  --namespace ome \
  -f values.yaml \
  --server-side=false

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --version 1.2.2 \
  --namespace ome \
  -f values.yaml

Helm installs the release and reports its status as deployed. Since v1.3, the first install fails with a webhook error: wait, then run the command again.

Pass the same values file to every upgrade: helm upgrade -f sets every value that isn't in the file back to the chart's default.

The model agent runs a pod on each node, including GPU nodes with the taint nvidia.com/gpu. Its priority class is system-node-critical, so Kubernetes can evict lower-priority pods to make room for it. To change where it runs, or what the chart installs, add these values to values.yaml and run the command again:

Helm values lists the model agent's other settings. Since v1.3, the chart also creates a ClusterServingRuntime named default-runtime, which OME uses only for InferenceServices that name it.

Step 4: Verify the installation¶

Wait for the controller and the model agent to be ready:

kubectl wait --for=condition=Available deployment/ome-controller-manager -n ome --timeout=5m
kubectl rollout status daemonset/ome-model-agent-daemonset -n ome --timeout=5m

deployment.apps/ome-controller-manager condition met
daemon set "ome-model-agent-daemonset" successfully rolled out

Then check which images they run:

kubectl get deployments,daemonsets -n ome \
  -o custom-columns='KIND:.kind,NAME:.metadata.name,IMAGE:.spec.template.spec.containers[0].image'

KIND         NAME                        IMAGE
Deployment   ome-controller-manager      ghcr.io/moirai-internal/ome-manager:v1.2.2
DaemonSet    ome-model-agent-daemonset   ghcr.io/moirai-internal/model-agent:v1.2.2

Since v1.3, the list also has the Deployment ome-prometheus, with the image docker.io/prom/prometheus:v3.0.1.

OME is ready to serve a model.

Optional components¶

Add these when you need their features. The controller looks for most of these components' CRDs only when it starts, so if you install one after OME, restart the controller with kubectl rollout restart deployment/ome-controller-manager -n ome.

The ome-scheduler, ome-alfred and ome-quota-manager charts need images that you build yourself. Each chart's guide shows how.

Install from source¶

To run a build of main, with the features marked Since v1.3, build and push OME's three images from a checkout, then install the charts from it. The build needs Docker and Go, and the OME agent image also needs a Rust toolchain: see Set up a development environment. From the checkout, run:

make push-manager-image push-model-agent-image push-ome-agent-image REGISTRY=registry.example.com/ome TAG=dev

The command builds the images for linux/amd64, and pushes ome-manager, model-agent and ome-agent to registry.example.com/ome with the tag dev. Set ARCH for another platform.

Install cert-manager and the PodMonitor CRD as in Step 1, then the CRDs from charts/ome-crd:

helm upgrade --install ome-crd ./charts/ome-crd --namespace ome --create-namespace

Helm installs the release and reports its status as deployed.

Point the ome-resources chart at your images in values.yaml:

global:
  hub: registry.example.com/ome
ome:
  controller:
    tag: dev
  omeAgent:
    tag: dev
modelAgent:
  enabled: true
  image:
    tag: dev

Then install OME from charts/ome-resources:

helm upgrade --install ome ./charts/ome-resources \
  --namespace ome \
  -f values.yaml \
  --server-side=false

helm upgrade --install ome ./charts/ome-resources \
  --namespace ome \
  -f values.yaml

The first install fails, because the webhook certificate isn't ready yet:

Release "ome" does not exist. Installing it now.
Error: Internal error occurred: failed calling webhook "clusterservingruntime.ome-webhook-server.validator": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block

Wait, then run the command again, as The first install fails with a webhook error shows. In helm list -n ome, both charts have the version 0.1.0, the placeholder version in a checkout.

Move a manifest install to the Helm charts¶

If you installed OME with make install or kubectl apply -k config/default, you can move it to the Helm charts and keep your InferenceServices, models and runtimes in place.

Troubleshooting¶

Helm reports resource mapping not found¶

If cert-manager or OME's CRDs are missing, installing the ome release fails before it creates anything, and names each object of an unknown kind:

Error: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "serving-cert" namespace: "ome" from "": no matches for kind "Certificate" in version "cert-manager.io/v1"
ensure CRDs are installed first, resource mapping not found for name: "default-runtime" namespace: "" from "": no matches for kind "ClusterServingRuntime" in version "ome.io/v1beta1"
ensure CRDs are installed first, resource mapping not found for name: "selfsigned-issuer" namespace: "ome" from "": no matches for kind "Issuer" in version "cert-manager.io/v1"
ensure CRDs are installed first]

Certificate and Issuer mean that cert-manager isn't installed: install it as in Step 1. ClusterServingRuntime means that OME's CRDs aren't installed: install them as in Step 2. Then run the command from Step 3 again.

The first install fails with a webhook errorUnreleased: coming in v1.3¶

The first install of the ome release fails, because the webhook certificate isn't ready yet:

Error: Internal error occurred: failed calling webhook "clusterservingruntime.ome-webhook-server.validator": could not get REST client: unable to load root certificates: unable to parse bytes as PEM block

Helm still creates the rest of the release. Wait for the certificate and the controller:

kubectl wait --for=condition=Ready certificate/serving-cert -n ome --timeout=5m
kubectl rollout status deployment/ome-controller-manager -n ome --timeout=5m

certificate.cert-manager.io/serving-cert condition met
deployment "ome-controller-manager" successfully rolled out

Then run the same helm upgrade --install command again. If it fails with the same error, run it once more. If it keeps failing, check that both releases are in the namespace ome.

An upgrade fails with a conflict¶

On Helm 4, helm upgrade can fail with a server-side apply conflict over the clientConfig.caBundle field of an OME webhook configuration. Run the upgrade again with --server-side=false, as in Step 3.

The controller or the model agent isn't ready¶

If the check in Step 4 times out, it prints:

error: timed out waiting for the condition on deployments/ome-controller-manager

Find the pod that isn't running with kubectl get pods -n ome, and read its events with kubectl describe pod -n ome, followed by the pod's name:

• A pod that stays Pending with Insufficient cpu or Insufficient memory needs more room than its node has. Keep the model agent off smaller nodes with modelAgent.gpuNodesOnly or modelAgent.nodeSelector, or lower modelAgent.resources.
• ErrImagePull or ImagePullBackOff means that the node can't pull the image. If your nodes pull from a mirror, see Install from a private registry.
• On v1.2.2, the controller's pods don't become ready when OME's CRDs are missing. Install them as in Step 2.

Uninstall¶

Delete your InferenceServices, models, accelerator classes and benchmark jobs before you uninstall OME. Only the controller removes their finalizers, so once it's gone, you can't delete them or OME's CRDs. Deleting a model also deletes its weights from the nodes. To keep them, first give the model the label models.ome/reserve-model-artifact: "true".

Check that none are left:

kubectl get inferenceservices,basemodels,clusterbasemodels,acceleratorclasses,benchmarkjobs --all-namespaces

No resources found

Since v1.3, add inferencereplicas,acceleratorquotas to the command: InferenceReplicas carry the finalizer ome.io/ir-teardown, and AcceleratorQuotas carry ome.io/accelerator-quota, which only the quota manager removes.

Then uninstall the ome release, and the ome-crd release after it:

helm uninstall ome -n ome
helm uninstall ome-crd -n ome

release "ome" uninstalled
release "ome-crd" uninstalled

Uninstalling ome-crd deletes OME's CRDs, and with them any OME objects left in the cluster. Check that the API server no longer serves them:

kubectl api-resources --api-group=ome.io

NAME   SHORTNAMES   APIVERSION   NAMESPACED   KIND

The namespace ome stays. Delete it once both releases are gone:

kubectl delete namespace ome

namespace "ome" deleted

cert-manager and the PodMonitor CRD stay installed, since other applications can use them.

Next steps¶

• Serve your first model: serve a small model and send it a request.
• Pre-configured models and runtimes: install ready-made models, runtimes and InferenceServices from OME's catalog.
• Serve a model on OMENative: run a model on OME's own workload controller. Since v1.3.
• Use the OME scheduler: place each gang of pods in one accelerator domain. Alpha, since v1.3.
• Run Alfred in recommend-only mode: let Alfred watch your GPU nodes and recommend migrations. Alpha, since v1.3.