Concepts
Learn how OME's parts fit together: its architecture, OMENative, scheduling, models, runtimes, serving, rollouts and traffic.
New to OME? Read How OME works first; the other pages build on it.
Architecture¶
How the OME manager, the model agent and OME's custom resources turn a model and a runtime into serving pods.
Deployment modes and OMENative
OME runs InferenceService components as Kubernetes Deployments or on OMENative, which creates, updates, repairs and moves each replica as one unit. Opt in to OMENative, see how OME picks a component's mode, and read what OMENative creates and reports.
OMENative¶
OMENative runs each replica of an InferenceService component as an Instance, one pod or a leader and its workers, that OME creates, updates, repairs, moves and removes as one unit. Multi-node engines and decoders use it by default, and any component can opt in.
An Instance is one replica of an OMENative component, a single pod or a leader and its workers, whose identity, phase, operation and failures OME records on the component's InferenceReplica.
Choose how an OMENative rollout replaces an Instance's pods, by starting new ones first, recreating them or updating them in place, and pace it with partition, maxSurge and maxUnavailable.
Choose whether OMENative repairs only a failed pod or rebuilds its whole Instance, and see how a restart runs.
OMENative moves an Instance to another node by starting its replacement first, and can run more or fewer Instances for a while without an edit to the InferenceService.
Scheduling and capacity¶
The OME scheduler and Alfred are alpha. Their configuration and behavior can change between releases. The scheduler needs Kubernetes 1.35.
With the scheduler-plugins PodGroup CRD installed, OME creates a PodGroup for each multi-pod OMENative Instance, so a gang-aware scheduler places its leader and workers together or not at all.
An alpha second scheduler that places each gang of pods in one accelerator domain, and binds its pods only when the whole gang fits.
Alfred, OME's alpha GPU cluster caretaker, recommends Instance moves when free GPUs fragment or a node turns unhealthy and, when you let it, asks OMENative to carry them out.
How Alfred's defragmentation and node-health policies find Instances to move, and how its arbiter and safety bounds decide which become recommendations or migration requests.
To divide a cluster's GPUs among teams with AcceleratorQuota, see Set accelerator quotas.
Models¶
BaseModel and ClusterBaseModel tell OME where a model's weights live and which nodes get them; OME parses the model's architecture, size and capabilities.
Serve weights fine-tuned from a base model, such as a LoRA adapter, from a zip file in OCI Object Storage.
Runtimes¶
A ServingRuntime or ClusterServingRuntime describes how to run a model server: its pods, and optionally the models it supports, so OME can choose it for an InferenceService.
Write one runtime per engine, and keep what differs for each model or GPU in small child runtimes that inherit from it.
Pin an InferenceService to a snapshot of its runtime, choose when runtime changes reach it, and roll back to an earlier snapshot.
An AcceleratorClass names one type of GPU and the nodes that have it, so an InferenceService can run its engine and decoder on that type, picked by name or by policy.
Serving¶
An InferenceService names a model, a serving runtime or both, and declares the engine, decoder and router components that OME turns into workloads.
Each InferenceService component can carry its own autoscaler block, which picks HPA, KEDA, External or None and the metrics or triggers to scale on.
An AutoscalerPolicy is an alpha, reusable set of KEDA or HPA autoscaling settings that InferenceService components share by name.
A BenchmarkJob runs genai-bench against an InferenceService with the traffic scenarios you choose, and stores the results.
Rollouts and traffic¶
Rollout groups roll an InferenceService's OMENative components out as one, so prefill and decode change together, by canary, blue-green or rolling update, with metric-gated canary steps.
A RolloutPolicy is an alpha, reusable canary, blue-green or rolling-update progression that InferenceService rollout groups attach by reference.
How clients outside the cluster reach an InferenceService: the Service that OME creates by default, or the Kubernetes Ingress or Gateway API HTTPRoutes it creates when you turn on ingress.
Set an InferenceService's load balancing, session affinity and endpoint override in spec.traffic. OME writes them into the gateway's backend policy and reports the result in status.
A TrafficMap is the alpha routing table that OME writes for a multi-cluster InferenceService: each workload cluster's share of the requests, and the reasons behind it.