InferenceService

An InferenceService serves a model. It names a base model, a serving runtime or both, and declares the components that serve the model: an engine, and optionally a decoder and a router. OME fills in each component from the runtime, runs it as a Deployment or on OMENative, and reports in status when the InferenceService is ready and where to send requests. A service can instead front replicas you run yourself: see Referenced replicas.

A minimal InferenceService¶

This InferenceService serves the ClusterBaseModel llama-3-2-1b-instruct with the ClusterServingRuntime vllm-llama-3-2-1b-instruct. Create both from config/models/meta/Llama-3.2-1B-Instruct.yaml and config/runtimes/vllm/llama-3-2-1b-instruct-rt.yaml in the OME repository. The model agent downloads the model to your nodes. It's off by default since v1.3: turn it on as Install OME shows, and set modelAgent.hostPath to /raid/models, where the models in config/models/ keep their weights. The model is gated, so the agent needs your Hugging Face token in the Secret hf-token in the ome namespace; see Credentials. Wait until the model is Ready, as Model lifecycle describes: until then, no runtime matches it, and the webhook rejects the InferenceService.

Create a namespace for it:

kubectl create namespace llama-demo

namespace/llama-demo created

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine: {}

Apply the file:

kubectl apply -f isvc.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-3-2-1b-instruct created

The empty engine section declares the engine, and the runtime fills it in. With no runtime named, OME picks one that supports the model, and the webhook's warning says which. If none does, the webhook rejects the InferenceService with a message that starts with no supporting runtime found for model llama-3-2-1b-instruct. The webhook rejects the InferenceService lists the other messages.

With the chart's default settings, OME creates:

• A Deployment and a Service named llama-3-2-1b-instruct-engine. The pods run only on nodes where the model is ready, as Node labels and status describes.
• A HorizontalPodAutoscaler and a PodDisruptionBudget of the same name. The autoscaler keeps CPU use at 80% with 1 to 3 replicas, and the budget lets one pod be unavailable at a time.
• A Service named llama-3-2-1b-instruct in front of the engine. It's how you reach the model while ingress creation is off, the default; see Ingress and external access.

Wait for the InferenceService to be ready:

kubectl wait --for=condition=Ready inferenceservice/llama-3-2-1b-instruct -n llama-demo --timeout=30m

inferenceservice.ome.io/llama-3-2-1b-instruct condition met

status.url is where to send requests. With ingress creation off, it's the address of the Service in front of the model:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo -o jsonpath='{.status.url}'

http://llama-3-2-1b-instruct.llama-demo.svc.cluster.local:8080

To send a request from outside the cluster, port-forward that Service, as Serve your first model does.

Model and runtime references¶

An InferenceService names a model, a runtime or both:

The webhook rejects an InferenceService that sets neither: at least one of spec.model or spec.runtime must be set. The one exception is a service over referenced replicas: it runs no model server of its own, and the replicas carry their own model and runtime.

The model¶

spec.model.kind defaults to ClusterBaseModel. To serve a BaseModel from the InferenceService's namespace, set kind: BaseModel:

spec:
  model:
    name: llama-3-2-1b-instruct
    kind: BaseModel

Leave kind out and OME needs a ClusterBaseModel of that name, even when a BaseModel in the namespace has it. When neither exists, the webhook rejects the InferenceService with a message that starts with referenced model "llama-3-2-1b-instruct" not found in namespace "llama-demo".

spec.model.fineTunedWeights names fine-tuned weights to serve on top of the model. OME can't serve a model with Sharded distribution in this release.

Since v1.3, spec.model.overlays mounts other models into every serving pod, for the runner to choose between at request time. Each overlay is at /opt/ml/model-overlays/<name>, and OVERLAY_<NAME>_MODEL_PATH holds that path, with the name upper-cased and its hyphens turned into underscores. OME doesn't switch models itself. If an overlay is disabled or deleted later, OME leaves it out, and the OverlaysReady condition says why.

The runtime¶

Since v1.3, you can leave out spec.model when the runtime loads the weights itself, from its container or from a volume or init container that you add under spec.engine. Such an InferenceService works without a BaseModel or the model agent. OME skips all model work: runtime selection, MODEL_PATH, the model volumes, the model-ready node selector and status.modelStatus. Overlays and fine-tuned weights need spec.model. A runtime whose arguments use $(MODEL_PATH), as the catalog runtimes do, gets that text literally unless you set MODEL_PATH in the runner's env. On v1.2.2, every InferenceService needs spec.model.

This InferenceService serves the same model without spec.model. Its vLLM engine downloads the weights from the Hugging Face repository that MODEL_PATH names, with the token in the hf-token Secret that Credentials creates in llama-demo:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-runtime-only
  namespace: llama-demo
spec:
  runtime:
    name: vllm-llama-3-2-1b-instruct
  engine:
    runner:
      env:
        - name: MODEL_PATH
          value: meta-llama/Llama-3.2-1B-Instruct
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token

A runtime you name can be a ClusterServingRuntime or a ServingRuntime in the InferenceService's namespace, as How OME resolves the name shows. Since v1.3, when that runtime doesn't declare support for the model, the webhook admits the InferenceService with a warning. It starts with runtime "<runtime>" does not declare support for model "<model>". If the runtime goes away later, OME leaves the running workload as it is and, since v1.3, sets the RuntimeReady condition to False until a runtime resolves.

By default, a change to the runtime rolls out to the InferenceService. To pin it to a snapshot of the runtime, set spec.runtime.autoSync: false, and optionally spec.runtime.revision: see Runtime revisions and pinning.

Components¶

An InferenceService has up to three components:

A component exists only when you declare it, or when spec.replicaRefs names a replica for its role. The runtime's engineConfig, decoderConfig and routerConfig fill in the components you declare, and your values win, so a runtime's routerConfig adds no router by itself.

Each component is a pod spec, so it takes pod fields such as nodeSelector and tolerations. It also has these fields:

One pod field has a default of its own: terminationGracePeriodSeconds, how long a pod gets to finish its in-flight requests after SIGTERM. A component whose InferenceService and runtime both leave it unset takes it from the deploy entry of the inferenceservice-config ConfigMap, in every deployment mode, on its leader and worker pod specs too. The ome-resources chart sets that default to 600 with ome.controller.terminationGracePeriodSeconds, so serving pods get 600 seconds. A manifest install sets none, and the pods take the Kubernetes default, 30 seconds. OME doesn't write the value into the InferenceService, so read it from the component's workload:

kubectl get deployment llama-3-2-1b-instruct-engine -n llama-demo \
  -o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}'

600

Deployment modes¶

Each component runs as a Deployment, the default, or, since v1.3, on OMENative. OMENative runs the component as an InferenceReplica, which manages each replica as one Instance: one pod, or a leader and its workers. OME creates, updates, repairs, moves and removes each Instance as one unit.

An engine or decoder with a leader or worker, set in the InferenceService or its runtime, runs on OMENative by default. Any component can opt in for per-Instance status, held revisions when updates keep failing, and updates that by default bring the new Instance up before the old one drains. Opt in the whole InferenceService with spec.deploymentMode: OMENative, or one component with its ome.io/deploymentMode annotation. MultiNode (deprecated), which only that annotation selects, runs a component as a LeaderWorkerSet and needs the LeaderWorkerSet controller.

Deployment modes and OMENative compares the modes and shows how OME resolves a component's mode.

Referenced replicas¶

Instead of declaring components, an InferenceService can front standalone InferenceReplicas: replicas you create directly, with no parent InferenceService, as Run a standalone InferenceReplica shows. spec.replicaRefs names, per role, the replica the service fronts. The service puts its stable names in front of the replicas' pods and reports the replicas in its status, and it writes nothing on them: their specs, rollouts and scaling stay with you.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-fronted
  namespace: llama-demo
spec:
  replicaRefs:
    engine:
      - llama-pool

This service fronts the standalone engine replica llama-pool: OME creates the stable Service llama-fronted-engine in front of the replica's pods, and routes requests to the service as it would to a declared engine, as Ingress and external access describes.

A referenced role runs on OMENative, and the replicas carry their own model and runtime, so such a service sets no other workload fields. The webhook enforces:

• engine is required; decoder and router are optional. Without an engine, the rejection starts with spec.replicaRefs.engine is required when spec.replicaRefs is set.
• Each role's list holds one name, and no replica is named for two roles.
• A service references all of its roles this way or none: no spec.engine, spec.decoder or spec.router next to spec.replicaRefs.
• No spec.model or spec.runtime — the model-or-runtime rule doesn't apply — and no spec.rollout, spec.scalingPolicy, spec.acceleratorSelector, spec.placement or spec.routing.
• spec.deploymentMode is unset or OMENative.

When you create the service, the webhook also reads each named replica. It must exist in the service's namespace — otherwise the rejection is spec.replicaRefs.engine names InferenceReplica llama-demo/llama-pool, which does not exist; create it first — be standalone, serve the role's component with its spec.component, and be fronted by no other InferenceService in the namespace: a replica has one service. Two services created at the same moment can both pass that last check; both then front the replica until you delete one of them.

spec.replicaRefs is fixed at create. The Services and the route are named for the service, so to front other replicas, delete the InferenceService and create a new one: the webhook rejects any change to the block with spec.replicaRefs is fixed at create; delete and recreate the InferenceService to front other replicas.

In status, each referenced role mirrors its replica: status.components.<component> names the replica in scaleTargetRef and carries its revisions and Instance counts, and EngineReady, DecoderReady and RouterReady follow those counts, so kubectl wait --for=condition=Ready works as usual. When a named replica is missing or wrong later — deleted, or no longer a standalone replica of the role's component — OME can't create what it doesn't own: it sets the role's ready condition to False with reason ReplicaRefMissing or ReplicaRefInvalid, records a Warning event with the same reason, and the role stops backing the route. It watches the replicas, so the service catches up on its own when the replica is back.

OME doesn't point the replicas at one another. A replica that needs a peer's address — an engine its decoder's, a router the engine's and the decoder's — needs the stable Service names, llama-fronted-engine and llama-fronted-decoder, in its own spec. They exist once the service does.

Multi-node serving¶

A model too large for one node's GPUs runs each replica across several pods. The engine or decoder declares them with leader and worker: each replica gets one leader pod and worker.size worker pods. The default worker.size is the runtime's value, or 1. A runner under leader or worker needs a name. Since v1.3, the webhook rejects a leader without a worker, or the reverse, with a message that starts with InvalidLeaderWorkerPairing.

On OMENative, the leader and its workers form one Instance and get OME_* environment variables, such as OME_LEADER_ADDRESS. When the PodGroup CRD is installed, OME also creates a PodGroup for each Instance, for gang scheduling. Serve a multi-node model walks through it.

The pre-configured multi-node runtimes read the LWS_* environment variables that a LeaderWorkerSet sets, so they need the deprecated MultiNode mode. To serve one as it is, set the component's ome.io/deploymentMode annotation to MultiNode, or make an OMENative copy of it. On v1.2.2, leader and worker select MultiNode: pin such a component before you upgrade, to keep it there.

Accelerator selection¶

When the runtime lists accelerator classes in acceleratorRequirements, OME can pick an accelerator class for the engine and the decoder. The class sets their node selector and, when the runner sets none, their GPU resources. Name a class in spec.acceleratorSelector or in a component's acceleratorOverride, or a policy, such as Cheapest, that picks one. Otherwise, OME picks no class. Select accelerators shows how.

ScalingUnreleased: coming in v1.3¶

Each component scales on its own, between its minReplicas and maxReplicas, by default with a HorizontalPodAutoscaler that keeps CPU use at 80%. The component's autoscaler block, which is alpha, can pick an HPA on your metrics, KEDA, or your own autoscaler or none instead. The block can also come from the runtime or from an AutoscalerPolicy, which is alpha and off by default; Which setting wins gives the order. A RawDeployment component can scale to zero with KEDA.

spec.scalingPolicy is alpha too. Its only working mode is Independent, the default; the webhook rejects Proportional and Pinned with ScalingModeNotImplemented. The kubectl ome autoscale command shows the autoscaling in effect for each component and where it came from.

On v1.2.2, kedaConfig, scaleTarget and scaleMetric configure scaling: Upgrade from v1.2.2 shows how to move them to autoscaler.

Rollouts and trafficUnreleased: coming in v1.3¶

On OMENative, related components, such as a prefill engine and its decoder, roll out together: spec.rollout puts them in up to three rollout groups. Each group rolls out a change by blue-green, rolling update or canary. Blue-green is the default. A canary moves capacity to the new revision in steps, with timed or manual pauses or Prometheus analysis, and failed analysis rolls it back to the stable revision. With the chart's settings, a revision whose updates keep failing is retried with backoff, then held until you release it or a newer revision replaces it.

A group can take its steps from a rollout policy through policyRef. Rollout policies are alpha and off by default: set ome.rolloutPolicy.enabled: true in both the ome-crd and ome-resources Helm charts, then restart the controller, as Turn on the feature describes.

Rollout groups and canaries need OMENative set in the InferenceService itself: spec.deploymentMode: OMENative, or the component's ome.io/deploymentMode annotation. A RawDeployment component, the default for a single-pod one, rolls out as a plain Deployment update, following its deploymentStrategy, with no group, canary or held revision.

kubectl ome rollout shows a rollout's progress and what holds it. Its alpha subcommands pause and resume a rollout, promote or roll back a canary, and repin a drifted plan.

spec.traffic sets load balancing, session affinity and endpoint override in your gateway's backend policy, an Envoy Gateway BackendTrafficPolicy or an Istio DestinationRule, as Traffic policy describes. It doesn't split requests between revisions.

spec.placement and spec.routing serve one InferenceService from several clusters. They're alpha, still in development and off by default. OME reads them only on a multi-cluster control plane, with ome.multicluster.enabled: true set in the ome-resources chart. See Traffic map.

Bounded lifetimeUnreleased: coming in v1.3¶

Give an InferenceService a bounded lifetime with spec.ttlSecondsAfterCreation: once that many seconds have passed since its creationTimestamp, OME deletes it, and garbage collection removes everything it owns. Unset, the default, means it lives until you delete it. The smallest TTL is one second.

This InferenceService serves the same model for a day, then goes away:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-experiment
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine: {}
  ttlSecondsAfterCreation: 86400

OME computes the deadline from creationTimestamp on every reconcile, so you can change the field on a live InferenceService: raising or lowering the value moves the deadline, and removing the field cancels it. The clock always counts from creation, never from the edit, so a TTL shorter than the InferenceService's age deletes it on the next reconcile.

When the deadline passes, OME deletes the InferenceService and records a Normal TTLExpired event: Deleted because ttlSecondsAfterCreation=86400 has passed. The delete names the expired object's UID as a precondition, so a recreated InferenceService with the same name is never deleted in its place: its own TTL, when it sets one, counts from its own creation.

OME checks the TTL before anything else in the reconcile, so a spec problem that stops reconciliation can't keep an expired InferenceService alive. While the deadline is ahead, each reconcile schedules the next no later than the deadline, so the delete needs no outside trigger. A copy that placement puts on a workload cluster ignores its own TTL: the source InferenceService's TTL governs it.

Status¶

The Ready condition is True when both EngineReady and IngressReady are. With ingress creation off, the default, IngressReady is True with reason IngressDisabled, so Ready follows the engine alone: check DecoderReady and RouterReady too. When IngressReady is True covers an Ingress and the Gateway API.

Rollout groups and traffic policies add their own conditions.

The RUNTIME column of kubectl get inferenceservice shows spec.runtime.name, so it's empty when OME picks the runtime. The engine's workload names the runtime in its serving-runtime label:

kubectl get deployment llama-3-2-1b-instruct-engine -n llama-demo -o jsonpath='{.metadata.labels.serving-runtime}'

vllm-llama-3-2-1b-instruct

status.modelStatus says whether the model loaded. It stays unset for an InferenceService without spec.model. Its transitionStatus is InProgress until the InferenceService is ready, then UpToDate. When the engine's main container fails, it's BlockedByFailedLoad, and lastFailureInfo holds the container's exit code and message.

status.components.<component> holds the component's url, once its workload is available. Since v1.3, it also reports the component's autoscaling and, on OMENative, its revisions and Instance counts and rollout phase and canary progress.

Next steps¶

• Deployment modes and OMENative: how each mode runs a component, and when to choose OMENative.
• Serve a model on OMENative: opt a single-pod InferenceService in, and check its InferenceReplica, Instance and pod.
• Serve a prefill-decode model: run prefill on the engine and decode on the decoder, each on OMENative.
• Component autoscaling: scale a component with an HPA, KEDA or your own autoscaler.
• Rollout groups: roll out changes to OMENative components together or in order.
• Reference a runtime explicitly and Troubleshoot runtime selection: name the runtime, or find out why OME can't pick one.
• Troubleshoot an InferenceService: find out why an InferenceService is rejected or isn't ready.