RUNTIME RECIPES
200+
Pre-defined for DeepSeek, Kimi, MiniMax, GLM, Qwen and Gemma
The Kubernetes operator for serving LLMs in production
Declare a model. OME picks the runtime, places it on the right GPUs, and lets you roll out changes as canaries.
ROLLOUT STRATEGIES
3
Rolling, blue-green and canary, across prefill and decode
KUBECTL OME
17
Commands to inspect, explain, and act
Works with
How it works
OMENative
Each replica is an Instance, a leader and its workers as one unit. The new one serves before the old one drains.
Canary rollouts
A share of prefill and decode moves together, step by step. Metrics promote the rest, or roll back.
Prefill and decode
Prefill in the engine, decode in the decoder. A router pairs them, and they roll out as one group.
Gang scheduling
A leader and its workers land together or not at all. With the PodGroup CRD, no half-started gangs.
The OME scheduler (alpha)
Packs a gang into one accelerator domain, all or nothing. For Kubernetes 1.35; pods opt in.
Alfred (alpha)
Spots fragmented GPUs and unhealthy nodes, recommends a move, and in execute mode makes room.
Runtime selection
Scores every runtime that supports the model and picks the best. Name one, and it's yours.
kubectl ome
Says what OME did and why: status, rollout, instance, runtime explain. Seventeen commands.
Any model The right runtime Progressive rollouts
Why OME?
OME turns models and GPUs into production endpoints on Kubernetes. You describe what to serve. OME chooses the runtime, generates the workloads, and rolls out your changes.
Adopters
Running OME in production? Add your organization to ADOPTERS.md.
OME creates, repairs, and migrates each replica as one Instance: one pod, or a leader and its workers. By default, new Instances serve before old ones drain. A failing revision is held, and prefill and decode can roll out together.
Read the OMENative overviewChoose your path
Start with what you need today, whether that's a first model on a test cluster or a fleet you already run.
Install OME and serve your first model.
New to OMEDeploy models, set up networking, roll out changes, and run OME itself.
GuidesHow models, runtimes, and InferenceServices fit together.
ConceptsThe OME API, kubectl ome commands, and matching rules.
Reference