Introduction
OME is a Kubernetes operator for serving large language models: it manages models and runtimes as resources, matches them, and runs the workloads.
OME, the Open Model Engine, is a Kubernetes operator for serving large language models. You create an InferenceService that names a model, a serving runtime or both. OME can put the weights on your nodes, or leave them to the runtime. It picks a runtime if you don't name one, and creates the workloads that serve the model.
Since v1.3, OME can run any component on OMENative, its own workload engine. OMENative creates, updates, repairs and moves each replica as one unit, and can roll out prefill and decode together. Components that span several nodes use it by default. Two optional, alpha add-ons look after the GPU cluster underneath. The OME scheduler schedules a group of pods together, and Alfred watches the GPU nodes and recommends migrations.
How OME serves a model¶
When OME manages the weights:
- You create a BaseModel or ClusterBaseModel that says where the weights live: Hugging Face, OCI Object Storage, the nodes' own disks or a PersistentVolumeClaim.
- The optional model agent, a DaemonSet, downloads the weights to the nodes you choose, or finds them there. It labels the nodes where the model is ready, and the serving pods run only on those nodes. By default, OME also reads the model's architecture, size and capabilities from its files. A model on a PersistentVolumeClaim skips the model agent: its serving pods mount the claim.
- You create an InferenceService that names the model. It can also name a serving runtime, the pod template for an engine such as SGLang or vLLM. If it doesn't, OME picks the runtime that best matches the model, among those that set
autoSelect: true. - OME builds the serving pods from the runtime's template, with the InferenceService's own settings taking precedence, and creates the workloads that run them. The deployment mode decides the kind of workload.
- By default, OME creates a Service named after the InferenceService, and you send requests to it. Turn on ingress creation, and OME creates an Ingress or Gateway API routes instead.
Since v1.3, you can instead skip steps 1 and 2 and name only a runtime, which loads the weights itself.
You install OME with Helm. The ome-crd chart installs OME's resource definitions, and the ome-resources chart runs the manager, OME's control plane. Since v1.3, ome-resources runs the model agent only when you set modelAgent.enabled=true, as Install OME does. The optional ome-serving chart deploys ready-made models, runtimes and InferenceServices from OME's catalog.
Deployment modes¶
An InferenceService has up to three components: an engine that runs the model, an optional decoder for prefill-decode disaggregation, and an optional router in front of them. Each component runs in a deployment mode:
| Mode | What runs the pods | When a component gets it |
|---|---|---|
| OMENative | Since v1.3. OME itself, which manages the pods of a replica as one unit. | By default when it has a leader and workers, or when you set deploymentMode: OMENative in the InferenceService's spec. |
| RawDeployment | A Deployment. | By default for other components. |
MultiNode (deprecated) |
A LeaderWorkerSet. | When the component's ome.io/deploymentMode annotation asks for it. On v1.2.2, by default when it has a leader and workers. |
To move a component off MultiNode, see Move from LeaderWorkerSet to OMENative.
Other features¶
| Feature | What you get |
|---|---|
| Accelerator classes | An InferenceService can pick a GPU type from its runtime's list, by name or with a policy such as Cheapest, and OME runs the model on that type's nodes. |
| Fine-tuned weights | Serve weights fine-tuned from a base model, such as a LoRA adapter. |
| Autoscaling | A component with no autoscaling settings gets a HorizontalPodAutoscaler that targets 80% average CPU utilization. |
| Runtime revisions | Pin an InferenceService to a snapshot of its runtime, and roll forward or back when you choose. |
| Rollout groups | Since v1.3. Roll out OMENative components, such as prefill and decode, as one group by canary, blue-green or rolling update. A canary can promote or roll back on Prometheus metrics. |
| Traffic policy | Since v1.3. Set how Envoy Gateway or Istio balances requests across an InferenceService's pods, for example by least request or by consistent hashing for session affinity. |
| Accelerator quotas | Since v1.3. Divide a cluster's GPUs among teams with a tree of AcceleratorQuotas. The quota manager, from the ome-quota-manager chart, turns the tree into Kueue cohorts and queues. |
kubectl ome |
Since v1.3. A kubectl plugin. status shows why an InferenceService is or isn't ready, and runtime explain shows which runtimes match its model, and why. |
Alpha featuresUnreleased: coming in v1.3¶
These features are alpha: their API and behavior can change between releases. Each stays unused until you install it, turn it on or set its field.
| Feature | What it does | How to turn it on |
|---|---|---|
| OME scheduler | A second scheduler that schedules the pods of a PodGroup together, and can pack them into one rack or other topology domain. | Install the ome-scheduler chart, which needs Kubernetes 1.35 and the scheduler-plugins PodGroup CRD. Set schedulerName: ome-scheduler on the runtime or the component. See Use the OME scheduler. |
| Alfred | Watches for fragmented GPU capacity and unhealthy nodes, and recommends migrations that fix them. | Install the ome-alfred chart. It starts no migration while alfredConfig.mode is recommend-only, the default. See Run Alfred in recommend-only mode. |
| Component autoscaler | The autoscaler block picks and configures a component's autoscaler: HPA, KEDA, External (one you run) or None. |
Set autoscaler on the component or on its runtime. The KEDA class needs KEDA installed. |
| Autoscaler policies | Reusable HPA or KEDA templates that components reference with autoscalerPolicyRef. |
Set ome.autoscalerPolicy.enabled: true in both the ome-crd and ome-resources charts. |
| Rollout policies | Reusable canary, blue-green or rolling-update progressions that rollout groups reference with policyRef. |
Set ome.rolloutPolicy.enabled: true in both the ome-crd and ome-resources charts. |
| Multi-cluster placement and routing | Still in development. A control-plane cluster places InferenceServices that set placement onto WorkloadClusters, and runs none itself. Routing adds a TrafficMap that tells a gateway how to split their traffic. |
On the control-plane cluster, set ome.multicluster.enabled: true and ome.multicluster.role: control-plane in the ome-resources chart. For routing, also set ome.multicluster.config.routing.enabled: true. |
The InferenceService fields placement, routing and scalingPolicy are alpha too, and so are the kubectl ome commands that change resources. See Guarded actions.
Next steps¶
- Install OME: install cert-manager and the OME charts, and check that OME is running.
- Serve your first model: serve a small model and send it a request.
- How OME works: how the manager, the model agent and OME's resources turn a model and a runtime into serving pods.
- OMENative overview: how OME runs each replica of a model as one Instance, and updates, repairs and migrates it.