OME
ome-projects/ome
  • v1.2.2
  • 516
  • 102
↑ ↓ to move ↵ to open esc to close See all results

How OME works

How the OME manager, the model agent and OME's custom resources turn a model and a runtime into serving pods.

OME serves large language models on Kubernetes. You describe what to serve and how in OME's custom resources. OME can bring the weights to your nodes or leave them to your runtime. It picks a runtime when you don't name one, and creates and manages the pods that serve the model. Since v1.3, OME can run any component on OMENative, its own workload engine, which manages each replica as one unit. Components that span several nodes use it by default.

Components

The manager is OME's core, and OMENative runs inside it. Two optional, alpha components work with OMENative: the OME scheduler places its gangs of pods, and Alfred recommends when to migrate its Instances.

Component What it does Where it runs
Manager OME's control plane. Its controllers turn OME's resources into serving workloads, and its webhooks check the resources and add containers and settings to serving pods. The Deployment ome-controller-manager, with three replicas by default, from the ome-resources chart
OMENative Since v1.3. OME's workload engine. It creates a component's pods itself and manages each replica as one Instance. Multi-node components use it by default, and any component can opt in. Inside the manager, on by default
Model agent Downloads the models that select its node, and labels the node when a model is ready there. The DaemonSet ome-model-agent-daemonset, from the ome-resources chart when you turn it on
ome-agent Downloads fine-tuned weights into the pods that serve them, and reads the config files of pvc:// models. An init container in serving pods, and Jobs
OME scheduler Since v1.3. Alpha. A second scheduler that places each gang of pods together in one accelerator domain. Pods opt in with schedulerName. It runs only on Kubernetes 1.35. A Deployment, ome-scheduler by default, from the ome-scheduler chart
Alfred Since v1.3. Alpha. The GPU cluster caretaker. It watches for fragmented GPU capacity and unhealthy nodes and, by default, only recommends Instance migrations. The Deployment ome-alfred, from the ome-alfred chart
Quota manager Since v1.3. Keeps the status of AcceleratorQuotas up to date, and can turn the quota tree into Kueue queues. A Deployment, ome-quota-manager by default, from the ome-quota-manager chart
kubectl ome Since v1.3. The OME CLI. It inspects OME's resources and runs alpha actions on them, with your Kubernetes identity. Your machine, as a kubectl plugin

Resources

All of OME's kinds have the API version ome.io/v1beta1. You write them all except TrafficMaps, which OME writes itself, and most InferenceReplicas: OME writes one for each OMENative component of an InferenceService, and you can also run a standalone one yourself.

Kind Scope Purpose
InferenceService Namespaced Serves a model. It names a model, a runtime or both, and describes the engine, decoder and router.
BaseModel, ClusterBaseModel Namespaced, Cluster A model: where its weights live and which nodes keep a copy.
FineTunedWeight Cluster Weights fine-tuned from a base model, which an InferenceService serves on top of that model.
ServingRuntime, ClusterServingRuntime Namespaced, Cluster How to serve a family of models: the models it supports, and the containers and settings of the engine, decoder and router.
AcceleratorClass Cluster A kind of GPU or other accelerator, and how to find the nodes that have it.
BenchmarkJob Namespaced A genai-bench run against an InferenceService or a URL.
AutoscalerPolicy Namespaced Since v1.3. Alpha, and off by default. An autoscaling template that InferenceService components attach by name.
RolloutPolicy Namespaced Since v1.3. Alpha, and off by default. A rollout progression that InferenceService rollout groups refer to.
InferenceReplica Namespaced Since v1.3. One pod set, run as Instances: the workload OME writes for one OMENative component, or a standalone pod set you create directly.
AcceleratorQuota Cluster Since v1.3. One entry in a tree of quotas that divides accelerators among teams.
WorkloadCluster Cluster Since v1.3. Alpha, and in development. A cluster that OME can place workloads onto, for multi-cluster serving, which is off by default.
TrafficMap Namespaced Since v1.3. Alpha, and in development. How the traffic of a multi-cluster InferenceService splits across clusters.

An InferenceService can use the BaseModels and ServingRuntimes in its own namespace, and every ClusterBaseModel and ClusterServingRuntime.

How weights reach the nodes

On each node that a BaseModel or ClusterBaseModel selects, the model agent:

  1. Downloads an hf:// or oci:// model from spec.storage.storageUri into the directory that spec.storage.path names, pulls a modelpack:// model into it through an llmman serve daemon, or finds a local:// model already there.
  2. Reads the model's config files. From them, OME fills in the model's architecture, size and other fields that runtime selection uses.
  3. Labels its node with the model's state there: Updating, then Ready or Failed. The label key names the model, such as models.ome.io/clusterbasemodel.qwen2-5-7b-instruct.

OME gives the engine and decoder pods a node selector on that label with the value Ready, so they run only on nodes that have the model. Until a node does, they stay Pending. Each pod mounts the model's directory read-only, and OME sets the MODEL_PATH environment variable to it. See Base models.

Since v1.3, the ome-resources chart installs the model agent only when you set modelAgent.enabled: true, as Install OME does. Without it, every model except a pvc:// model stays In_Transit, and the pods that serve it stay Pending.

A pvc:// model stays on its PersistentVolumeClaim, and the serving pods mount the claim. See Serve models from a PVC.

A fine-tuned weight downloads into the engine and decoder pods, through an init container that runs ome-agent.

Since v1.3, an InferenceService can skip this path and name only a runtime, which loads the weights itself.

From InferenceService to pods

An InferenceService serves one model with up to three components:

  • the engine, which runs the model;
  • optionally, a decoder, for prefill-decode disaggregation: the engine's pods run the prefill phase, and the decoder's pods the decode phase;
  • optionally, a router in front of them, which sends requests to their pods.

When you create or change an InferenceService, OME works through these steps:

  1. Find the model that spec.model names. The kind defaults to ClusterBaseModel, so an InferenceService that uses a BaseModel sets spec.model.kind: BaseModel.
  2. Choose the runtime. When spec.runtime names a runtime, OME uses it. Otherwise, it picks the best match among the runtimes that support the model and allow automatic selection. See How runtimes match the model. By default, the InferenceService follows changes to its runtime. To pin it to one revision, see Runtime revisions and pinning.
  3. Build the components. OME starts from the runtime's engineConfig, decoderConfig and routerConfig, and the InferenceService's own settings win. When the runtime lists accelerator classes and the InferenceService names a class or a selection policy, OME picks an AcceleratorClass for the engine and decoder. Their pods get the class's node selector. See Select accelerators.
  4. Create each component's workload for its deployment mode, and a Service for it. Both are named after the InferenceService and the component, such as llama-engine for the engine of llama.
  5. Expose the InferenceService. By default, OME creates a ClusterIP Service named after the InferenceService, which sends requests to the router's pods, or to the engine's when there's no router. When you turn on ingress creation, OME creates a Kubernetes Ingress or Gateway API HTTPRoutes instead. See Ingress and external access.
  6. Report status. OME records the InferenceService's URL, the state of its components, and conditions such as Ready in its status. Since v1.3, kubectl ome status shows in one report why the InferenceService is or isn't ready.

If OME can't find a runtime, for example because the runtime was deleted, it records a RuntimeNotFound warning event and leaves the serving workload as it is.

Deployment modes and OMENative

Since v1.3, an engine or decoder with a leader or worker block runs on OMENative by default. The block can come from the InferenceService or its runtime. Any other component runs as a RawDeployment. To run all of an InferenceService's components on OMENative, the router included, set spec.deploymentMode: OMENative.

Text
OMENative:      InferenceService -> InferenceReplica -> pods, grouped into Instances
RawDeployment:  InferenceService -> Deployment -> ReplicaSet -> pods

On OMENative, each replica of a component is an Instance: one pod, or a leader pod and its worker pods for a model that spans nodes. By default, when a pod of a multi-pod Instance fails, OME recreates all of its pods. When the scheduler-plugins PodGroup CRD is installed, OME also creates a PodGroup for each multi-pod Instance, so that a gang-aware scheduler, such as the OME scheduler, places its pods together. See Gang scheduling.

A component's ome.io/deploymentMode annotation wins over all of this. Since v1.3, it's the only way to run a component in the deprecated MultiNode mode, on a LeaderWorkerSet. On v1.2.2, a leader or worker block makes a component MultiNode, and there's no OMENative mode. See How OME resolves the mode.

Next steps