Run a standalone InferenceReplica Unreleased: coming in v1.3

An InferenceReplica is one pod set on OMENative: N Instances of the same pod templates, with one revision history and one rollout. OME writes one for each OMENative component of an InferenceService. You can also create one directly, with no parent InferenceService: a standalone replica, whose spec, scaling and rollouts stay with you. Run one as a serving pool you manage yourself, or put an InferenceService's stable names and routes in front of it with referenced replicas.

Here you create a standalone engine replica that serves a model, check what OME runs for it, and scale it.

One template source¶

A replica renders its pods from exactly one template source. The webhook rejects a spec that sets none, or runners next to a reference:

With modelRef or runtimeRef, the controller renders the pod templates in memory on every pass and never stores them in your spec. The live model and runtime render, so there's no runtime pinning: the webhook rejects spec.runtimeRef.autoSync: false and spec.runtimeRef.revision. The runtime's pod spec places the pods — node selector, affinity and resources come from it — so a second accelerator pool is a second runtime.

Step 1: Create the replica¶

Create a namespace:

kubectl create namespace llama-demo

namespace/llama-demo created

Save this InferenceReplica as llama-pool.yaml. It serves the model with the runtime OME selects for it, as one engine Instance:

apiVersion: ome.io/v1beta1
kind: InferenceReplica
metadata:
  name: llama-pool
  namespace: llama-demo
spec:
  component: engine
  modelRef:
    name: llama-3-2-1b-instruct
  replicas: 1

spec.component is the role the pod set fills — engine, decoder or router — and is immutable: to change it, recreate the replica. spec.replicas counts Instances. The name becomes the prefix of the replica's pod and Service names, so it must match [a-z]([-a-z0-9]*[a-z0-9])?, and it must not collide with an InferenceService: see What the webhook rejects.

Create the replica:

kubectl apply -f llama-pool.yaml

inferencereplica.ome.io/llama-pool created

Wait for it to be ready. Pulling the vLLM image and loading the model can take several minutes:

kubectl wait --for=condition=Ready inferencereplica/llama-pool -n llama-demo --timeout=30m

inferencereplica.ome.io/llama-pool condition met

Step 2: Check what OME runs¶

List the InferenceReplicas, irep for short. The columns count Instances:

kubectl get inferencereplicas -n llama-demo

NAME         COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
llama-pool   engine      1         1         1       1           8m

kubectl ome get inferencereplicas shows more, including a PARENT column that's - for a standalone replica.

The pods are named {replica}-{component}-{index}-{runner}-{ordinal}, and their ome.io/inferenceservice label carries the replica's own name, since there's no InferenceService. List the pod with its ome.io/serving readiness gate, which is True while the pod is in rotation:

kubectl get pods -n llama-demo -l ome.io/inferenceservice=llama-pool \
  -o 'custom-columns=NAME:.metadata.name,SERVING:.status.conditions[?(@.type=="ome.io/serving")].status'

NAME                            SERVING
llama-pool-engine-0-default-0   True

To read the pod's logs, pass the replica's name to kubectl ome logs: the command selects pods by this label and never reads an InferenceService, so kubectl ome logs llama-pool -n llama-demo prints them, and -f follows.

OME also creates the component's headless Service, which the rotation and the pods' DNS use, and PodGroups for multi-pod Instances when the PodGroup CRD is installed, for gang scheduling:

kubectl get services -n llama-demo -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,CLUSTER-IP:.spec.clusterIP

NAME                         TYPE        CLUSTER-IP
llama-pool-engine-headless   ClusterIP   None

That's all: the stable Service, the route, the PodDisruptionBudget and the autoscaler of an InferenceService component come from the InferenceService, which a standalone replica doesn't have. To get stable names and routing in front of the pods, front the replica with an InferenceService; to reach them directly, create your own Service over the ome.io/inferenceservice=llama-pool label.

Step 3: Scale the replica¶

Whatever targets the replica's scale subresource writes spec.replicas. Scale it by hand:

kubectl scale inferencereplica llama-pool -n llama-demo --replicas=2

inferencereplica.ome.io/llama-pool scaled

The new Instance takes the lowest free index, here 1:

kubectl get inferencereplica llama-pool -n llama-demo

NAME         COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
llama-pool   engine      2         2         2       2           12m

To autoscale, point your own HorizontalPodAutoscaler or KEDA ScaledObject at the scale subresource, with a scaleTargetRef of apiVersion: ome.io/v1beta1, kind: InferenceReplica and the replica's name, as Bring your own autoscaler shows for an InferenceService component. status.labelSelector serves the HPA's pod lookup. Only the InferenceService controller turns spec.autoscaler into an HPA or a ScaledObject, so on a standalone replica the webhook rejects the HPA and KEDA classes; an External scaler targets the scale subresource directly, so that class is accepted.

How updates roll out¶

A change that alters the rendered pod templates mints a new revision, and the replica rolls its Instances to it by spec.lifecycle.updateStrategy — SurgeThenDrain when unset, which brings each Instance's new pods up and serving before the old ones drain. Because a modelRef or runtimeRef replica renders from the live objects, a change to the referenced model or runtime rolls the replica too: OME watches them, and there's no pinning. On a runners replica, you apply the new templates yourself.

You hold and steer a rollout with the same fields the InferenceService controller writes on the replicas it projects:

• spec.paused: true stops updates, new Instances and migrations; spec.pauseMode: Freeze also stops Instance repair. In-flight operations resume on unpause.
• spec.pacing.partition keeps that many of the lowest-indexed Instances on their current revision; it must not exceed spec.replicas.
• spec.pacing.rollbackToRevision rolls every Instance back to a named ControllerRevision of this replica.
• spec.minReadySeconds makes a newly Ready pod wait before it counts as Available, which paces drains and promotions. On a replica it's this top-level field: the webhook rejects the nested spec.lifecycle.minReadySeconds, which nothing reads here.

Failed Instances work as on any OMENative component: set the ome.io/reset-instances annotation on the replica to rebuild them once you've fixed the cause. Manual Instance migration isn't available: it surges through a per-revision Service that only an InferenceService creates, so the webhook rejects a migration request on a standalone replica.

What the webhook rejects¶

OME's validating webhook checks every InferenceReplica create and update. It tells the forms apart by the InferenceService controller owner reference and spec.parentRef: a replica with either is projected, and rejects your spec writes with a message that ends in edit the InferenceService instead. On a standalone replica, which has neither, it rejects:

The webhook also checks every writer's shape: an autoscaler block that doesn't validate, spec.pacing.partition above spec.replicas, and a runner template whose volumeMounts name a volume the pod doesn't declare. Deleting a replica is never gated.

Troubleshooting¶

The replica is not Ready¶

When the replica has no pod templates it can run, its Ready condition is False and a Warning event carries the same reason and message. Read them with kubectl describe inferencereplica llama-pool -n llama-demo:

OME leaves the replica's existing pods as they are, and watches the referenced model and runtime: fix the referenced object, and the replica renders without you touching it. For an Instance that fails after rendering, such as an image that can't be pulled, see The Instance fails: a standalone replica retries and holds revisions the same way.

A RuntimeCompatibilityAdvisory warning¶

When spec.runtimeRef names a runtime that doesn't declare support for the model in spec.modelRef, the replica still renders, and OME records a Warning event that starts with Runtime <runtime> does not declare support for model <model> and ends with proceeding because the runtime was named explicitly. Serving may still fail at load time: check the runtime's supportedModelFormats against the model, or drop spec.runtimeRef and let OME select a supporting runtime.

Clean up¶

Delete the replica and the namespace. OME drains the Instances before it deletes their pods:

kubectl delete inferencereplica llama-pool -n llama-demo
kubectl delete namespace llama-demo

inferencereplica.ome.io "llama-pool" deleted from llama-demo namespace
namespace "llama-demo" deleted

If a pod stays Terminating, such as on a node that's gone, see Recover stuck deletions. The model and the runtime are cluster-scoped and stay.

Next steps¶

• Referenced replicas: front standalone replicas with an InferenceService's stable names, routing and status.
• Instances: the phases, operations and incarnations of the Instances in your replica.
• OMENative update strategies: how each strategy replaces an Instance's pods, and when to pick which.
• Bring your own autoscaler: an HPA on the scale subresource that OME publishes.
• kubectl ome get: list replicas with their parent, revisions and lifecycle condition.