Guides

Deploy an InferenceService

Write the smallest InferenceService that serves a model, with only a model, a model and a runtime, or only a runtime, then wait for it, send it a request and see what OME created.

Reference a runtime explicitly

Name the ServingRuntime or ClusterServingRuntime that an InferenceService uses instead of letting OME choose one, then check that OME used it.

Troubleshoot runtime selection

Find out why OME can't auto-select a runtime for your model, from kubectl's error or the RuntimeReady condition, then fix the runtime or name one.

Serve models from a PVC

Serve model weights that already live on a PersistentVolumeClaim by pointing a BaseModel at a pvc:// URI, with no download to nodes.

Serve models from node-local storage

Serve model weights already on your nodes' disks with a local:// storage URI, which OME validates in place and mounts read-only.

Select accelerators

Pick the AcceleratorClass for the engine and decoder by naming a class, or let OME choose one with the BestFit, Cheapest, MostCapable or FirstAvailable policy.

Configure pod disruption budgets

Control how many of a component's pods a node drain can take down, with a budget on each component or defaults for each deployment mode.

Run benchmarks

Run a BenchmarkJob against an InferenceService with realistic traffic scenarios, and collect its throughput and latency results from storage.

Serve a model on OMENative

Opt a single-pod InferenceService into OMENative with spec.deploymentMode, then check the InferenceReplica, the Instance and the pod that OME creates.

Serve a multi-node model

Run one copy of a model across several nodes by giving its engine a leader and workers, which OMENative creates, updates and repairs together as one Instance.

Serve a prefill-decode model

Serve a model with prefill on the engine and decode on the decoder, each with its own InferenceReplica on OMENative, and scale prefill on its own.

Run a standalone InferenceReplica

Create an InferenceReplica directly, with no parent InferenceService: give it a model, a runtime or rendered pod templates, then scale, update and pause the pod set yourself.

Provision TPU slices on demand

Serve an OMENative component on GKE TPU node pools that provision slices on demand: OME creates a Slice for each Instance's pods, holds the pods until it's ready, and releases it when no pod needs it.

Move from LeaderWorkerSet to OMENative

Move a multi-node InferenceService off the deprecated MultiNode mode, which runs on LeaderWorkerSet, onto OMENative, and delete the LeaderWorkerSet that OME leaves behind.

Spread Instances across fault domains

Spread an OMENative engine's or decoder's Instances across racks, fabric slices or zones with topologySpread and topologySpreadKey, as a preference or a requirement.

Set Instance readiness deadlines

Bound how long OMENative waits for a new Instance to become Ready with instanceReadyTimeout and the stuck-pod and unschedulable grace periods, and see what OME does when one runs out.

Reset failed Instances

Rebuild the OMENative Instances that OME marked Failed, once you've fixed the cause, with the ome.io/reset-instances annotation.

Recover stuck deletions

Recover OMENative pods stuck Terminating on dead nodes with the opt-in forceDelete escalation, and bound how long a deleting InferenceReplica holds its teardown finalizer.

Change the Instance status encoding

Choose how OME stores per-Instance status on InferenceReplicas, as compact ColumnarV2 columns or a DenseV1 list, and switch a cluster between them.

Configure ingress

Configure how OME exposes InferenceServices through Kubernetes Ingress or Gateway API with the ingress block of the inferenceservice-config ConfigMap.

Expose a service without ingress

Put an InferenceService behind a LoadBalancer or NodePort Service with the ome.io/service-type annotation, without an Ingress controller or a Gateway.

Choose a Gateway API host scheme

Choose between one shared hostname with per-service path prefixes and a subdomain per InferenceService for the HTTPRoutes OME generates.

Use multiple gateways

Attach OME's HTTPRoutes to more than one Gateway, such as an internal and an external one, and read the gateways' addresses in status.

Use per-namespace gateways

Route each namespace's InferenceServices through that namespace's own Gateway, and override the gateway for a single service with annotations.

Configure route timeouts

Set the request timeout on OME's HTTPRoutes for one InferenceService or the whole cluster, or turn it off so long generations can finish.

Set appProtocol on services

Set Kubernetes appProtocol values on the Service ports OME generates, so gateways and service meshes use the right protocol for your model server.

Pause and resume a rollout

Pause in-flight rollout work on an OMENative InferenceService with the alpha kubectl ome rollout pause action, and continue it with resume.

Promote or roll back a canary

Advance a gated canary step with kubectl ome rollout promote, or abort the canary back to the stable revisions with rollback; both are alpha.

Release a held revision

Get OME to retry a revision it stopped retrying, once you've fixed what made it fail, with the alpha kubectl ome instance release-held command.

Repin a drifted rollout plan

Apply an edit to spec.rollout or a RolloutPolicy to the active rollout run, keeping its progress, with the alpha kubectl ome rollout repin action.

Pace rollouts with minReadySeconds

Make a new OMENative pod stay Ready for a warm-up window before it counts as Available, so a rollout waits before draining the old pods.

Request a transient scale

Change how many Instances one OMENative component runs with the alpha kubectl ome scale command, until its autoscaler or OME sets the count again.

Request an Instance migration

Move one OMENative Instance to another node, with its replacement serving before the old one drains, using the alpha kubectl ome migration start command.

Scale to zero with KEDA

Scale an idle RawDeployment component to zero replicas with a KEDA autoscaler, and back up when a trigger becomes active.

Bring your own autoscaler

Turn off OME's autoscaler for a component with the External class, and scale it with your own autoscaler through the scaleTargetRef that OME publishes.

Use the OME scheduler

Install the alpha ome-scheduler as a second scheduler and opt workloads into topology-packed gang placement; it requires Kubernetes 1.35.

Run Alfred in recommend-only mode

Install Alfred and read the Instance moves it recommends when free GPUs fragment or a node turns unhealthy, then tune its policies.

Set up scheduler simulation for Alfred

Load your schedulers' configurations into Alfred, so that it predicts where the replacement pods of each move would land.

Let Alfred migrate Instances

Turn on Alfred's execute mode, so that it asks OMENative to migrate Instances off fragmented or unhealthy nodes within the limits you set.

Set accelerator quotas

Install the ome-quota-manager, split a cluster's GPUs among teams with an AcceleratorQuota tree, and charge workloads to each team's Kueue budget.

Publish a global endpoint

Give an alpha multi-cluster InferenceService one global hostname that splits its traffic across workload clusters by its TrafficMap's weights.

Configure routing health probes

Configure the alpha end-to-end probe that sets a workload cluster's TrafficMap weight to zero after repeated failures, for every InferenceService or for one.

Drain a workload cluster

Use the alpha kubectl ome traffic drain to take a workload cluster out of an InferenceService's traffic for maintenance, and undrain to give it back.

Weight traffic for heterogeneous clusters

Give a workload cluster whose replicas serve more requests a larger share of an alpha multi-cluster InferenceService's traffic, with capacity factors in spec.routing.capacityFactors.

Tune routing capacity polling

Cap a workload cluster's TrafficMap weight at the replicas that its endpoint reports it can serve, with an alpha poll set in the chart or for one InferenceService.

Configure the controller

Tune the OME controller manager's command-line flags, including reconcile concurrency, leader election and runtime-revision garbage collection.

Set replica defaults

Choose the minReplicas and maxReplicas that components get when the InferenceService and its runtime leave them unset, with the ome-resources chart's ome.controller.replicas values.

Run the model agent

Turn on the optional model agent for models stored on the nodes, and configure how it prepares them.

Keep downloaded model files

Keep a model's downloaded files on the nodes after you delete the model, with the models.ome/reserve-model-artifact label.

Share Hugging Face artifacts

Keep one downloaded copy of a Hugging Face snapshot per node and share it across BaseModels and ClusterBaseModels with downloadPolicy ReuseIfExists.

Replicate model weights

Copy model weights between Hugging Face, OCI Object Storage and PVCs with the ome-agent replica command, run as a one-shot Kubernetes Job.

Collect metrics

Tune or turn off the short-retention Prometheus that the ome-resources chart deploys for KEDA autoscaling and canary analysis, or scrape OME with your own.

Set up alerting

Enable the ome-resources chart's five P0 PrometheusRule alerts, which are off by default, and tune their thresholds and selectors.

Move a manifest install to the Helm charts

Move an OME install from make install or kubectl apply -k to the ome-crd and ome-resources Helm charts, and keep your InferenceServices, models and runtimes in place.

Troubleshoot an InferenceService

Find out why an InferenceService isn't ready or serving by checking its status, model, runtime and logs, then look up the fix for what you see.