Component autoscaling Unreleased: coming in v1.3

An autoscaler block lets each component of an InferenceService scale on the signal that suits it, from CPU utilization to the requests waiting in vLLM's queue. OME can create a HorizontalPodAutoscaler or a KEDA ScaledObject for it, leave it to an autoscaler that you run, or hold it at a fixed count. A component with no autoscaling settings gets a HorizontalPodAutoscaler that targets 80% average CPU utilization. The autoscaler block and the autoscaling fields in status are alpha, and so are AutoscalerPolicy and spec.scalingPolicy. Their fields and behavior can change between releases, but the block needs no feature flag.

The examples use the llama-demo namespace and the llama-3-2-1b-instruct InferenceService from A minimal InferenceService, and assume OME runs in the ome namespace.

Classes¶

Put the block in spec.engine, spec.decoder or spec.router. Its class decides what OME creates, and who sets the replica count:

HPA¶

This version of isvc.yaml scales the engine between 1 and 4 replicas to keep its average CPU utilization at 70%:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 4
    autoscaler:
      class: HPA
      hpa:
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 70

OME copies hpa.metrics and hpa.behavior into the HorizontalPodAutoscaler as they are. Without metrics, the default CPU target applies.

Apply the file:

kubectl apply -f isvc.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-3-2-1b-instruct configured

Check the engine's autoscaling in the InferenceService's status:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='CLASS:.status.components.engine.autoscaler.class,MANAGED-BY:.status.components.engine.autoscaler.managedBy,SOURCE:.status.components.engine.autoscaler.specSource,CURRENT:.status.components.engine.autoscaler.currentReplicas,DESIRED:.status.components.engine.autoscaler.desiredReplicas,KIND:.status.components.engine.scaleTargetRef.kind,TARGET:.status.components.engine.scaleTargetRef.name'

CLASS   MANAGED-BY   SOURCE   CURRENT   DESIRED   KIND         TARGET
HPA     ome          isvc     1         1         Deployment   llama-3-2-1b-instruct-engine

OME's HorizontalPodAutoscaler now scales the engine's Deployment, which runs 1 replica. Read the result in status explains each column.

KEDA¶

class: KEDA needs KEDA in the cluster. Without it, the InferenceService's ingress and status stop updating, as KEDA isn't installed describes. The controller looks for KEDA when it starts, so if you install KEDA after OME, restart the controller:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

This version of isvc.yaml scales the engine on the requests waiting in vLLM's queue. The trigger reads them from ome-prometheus, the Prometheus server that the ome-resources chart deploys:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 4
    autoscaler:
      class: KEDA
      keda:
        pollingInterval: 15
        cooldownPeriod: 300
        triggers:
          - type: prometheus
            metricType: AverageValue
            metadata:
              serverAddress: http://ome-prometheus.ome.svc:9090
              query: 'sum(vllm:num_requests_waiting{namespace="llama-demo",inferenceservice="llama-3-2-1b-instruct",component="engine"})'
              threshold: "5"

The query sums the waiting requests across the engine's pods. With metricType: AverageValue, KEDA divides the sum by the replica count, so it adds replicas to keep about 5 waiting requests per replica. Collect metrics describes the Prometheus server and the labels it adds.

The engine already has a HorizontalPodAutoscaler from the HPA example, so apply this file in two steps, as Switch from HPA to KEDA shows.

OME copies the keda fields into the ScaledObject as they are:

Switch from HPA to KEDA¶

OME creates the new autoscaler before it deletes the old one. A default KEDA install rejects a ScaledObject for a workload that a HorizontalPodAutoscaler already scales. So move a component from OME's HorizontalPodAutoscaler to KEDA in two steps. That includes a running component with no autoscaling settings, which has OME's default HorizontalPodAutoscaler.

First, set class: External, keep minReplicas at 1 or more, and apply the file. OME deletes its HorizontalPodAutoscaler and keeps the replica count. Check that the HorizontalPodAutoscaler is gone:

kubectl get hpa llama-3-2-1b-instruct-engine -n llama-demo

Error from server (NotFound): horizontalpodautoscalers.autoscaling "llama-3-2-1b-instruct-engine" not found

Then apply the KEDA version of isvc.yaml. Check the ScaledObject shows how to confirm what it scales.

A switch in one step fails, whether it comes from the InferenceService, its runtime or an AutoscalerPolicy. OME's HorizontalPodAutoscaler keeps scaling the component, and the InferenceService's ingress and status stop updating. The controller logs Failed to reconcile component, with an error that contains is already managed by the hpa:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

To recover, delete OME's HorizontalPodAutoscaler:

kubectl delete hpa llama-3-2-1b-instruct-engine -n llama-demo

horizontalpodautoscaler.autoscaling "llama-3-2-1b-instruct-engine" deleted from llama-demo namespace

OME then creates the ScaledObject when it retries.

External and None¶

Use External to scale a component with an autoscaler that you run, pointed at the object in status.components.<component>.scaleTargetRef. Bring your own autoscaler shows how. A switch to External or None deletes the autoscaler that OME created. With External, OME keeps the current replica count until your autoscaler changes it.

If you already run a HorizontalPodAutoscaler or ScaledObject under the name that OME would use, HPA and KEDA fail with an error that contains is not controlled by expected owner. Delete that object, or keep it with class: External.

What the autoscaler scales¶

The autoscaler's target depends on the component's deployment mode:

On OMENative, a Resource or Pods metric averages over every pod in every Instance, workers included. Request a transient scale says how long a manual change to an OMENative component's count lasts under each class.

Replica bounds¶

minReplicas and maxReplicas sit next to autoscaler in the component, and OME passes them to the autoscaler that it creates. A bound that the InferenceService and its runtime leave unset comes from the replica defaults. The ome-resources chart sets them to 1 to 3 replicas for the engine and the decoder, and 1 to 2 for the router. OME raises a default maximum that's below minReplicas to match.

Keep minReplicas at or below maxReplicas, wherever each is set. Admission checks the pair only when the InferenceService sets both, so a minReplicas above the runtime's maxReplicas gets through. OME then raises an OMENative component's maximum to match. On a RawDeployment component, the controller logs minReplicas must not exceed maxReplicas, and the autoscaler and the InferenceService's status stop updating. Set replica defaults shows how to find the error.

A RawDeployment component scales to zero with minReplicas: 0 and a class: KEDA block in the InferenceService itself, as Scale to zero with KEDA shows. An OMENative component keeps at least one Instance, even with minReplicas: 0: see OMENative components can't scale to zero.

Which setting wins¶

OME takes a component's autoscaling block from the first of these places that has one:

The block comes whole from one place, so a class: HPA block without hpa gets the default CPU target, even when the runtime lists metrics.

AutoscalerPolicy is off by default: Turn on the feature shows how to turn it on. If OME can't use a policy reference, the component keeps its last autoscaler, instead of falling through to the runtime or the default: see Fail-closed behavior.

Read the result in status¶

OME reports each component's autoscaling in status.components.<component>.autoscaler:

status.components.<component>.scaleTargetRef names the scaled object by apiVersion, kind and name. kubectl ome autoscale shows these fields for every component, and explains where each setting came from.

What admission rejects¶

The InferenceService webhook rejects the following, with the component and the reason in its message, as in engine: KedaTriggersRequired: class=keda requires at least 1 trigger:

The ServingRuntime and ClusterServingRuntime webhooks check the blocks in engineConfig, decoderConfig and routerConfig the same way, and reject an autoscalerPolicyRef in a runtime. Legacy autoscaling lists the legacy annotations' values, and Attach a policy to an InferenceService covers policy references.

Upgrade from v1.2.2¶

v1.3 removes the component's scaleMetric and scaleTarget, and the InferenceService's spec.kedaConfig, as Removed in v1.3 lists. Move them into an autoscaler block:

v1.3 copies query as it is, so replace a %s from customPromQuery with the component's name, such as llama-3-2-1b-instruct-engine.

Next steps¶

• Autoscaler policy: share one set of autoscaling settings across InferenceServices.
• Scale to zero with KEDA: scale an idle RawDeployment component down to no replicas.
• Bring your own autoscaler: scale a component with an autoscaler that you run, through class: External.
• Request a transient scale: ask once for a different replica count on an OMENative component.
• kubectl ome autoscale: show the autoscaling in effect for each component.
• ComponentAutoscaler: every field of the autoscaler block, in the API reference.
• Deployment modes and OMENative: how each mode runs a component.