Autoscaler policy Unreleased: coming in v1.3

An AutoscalerPolicy lets the InferenceServices in a namespace share one set of autoscaling settings: KEDA triggers, or HorizontalPodAutoscaler metrics and behavior. An InferenceService component names the policy in autoscalerPolicyRef, and OME builds its autoscaler from the policy, filling in the component's names and replica bounds. Edit the policy, and OME updates every component that uses it. AutoscalerPolicy is alpha and off by default: Turn on the feature shows how to turn it on.

Use a policy when several InferenceServices should scale the same way. A runtime can also carry autoscaling settings for the InferenceServices that use it, and a policy outranks them.

The examples use the llama-demo namespace and the llama-3-2-1b-instruct InferenceService from A minimal InferenceService, and assume OME runs in the ome namespace.

Turn on the feature¶

Set ome.autoscalerPolicy.enabled: true in both the ome-crd and ome-resources charts, version 1.3.0 or later. A prometheus trigger also needs a metric provider, and the chart binds none by default. This values file turns the feature on, and binds the examples' provider, cluster-prometheus, to the bundled Prometheus:

ome:
  autoscalerPolicy:
    enabled: true
  metricProviders:
    cluster-prometheus:
      serverAddress: "http://ome-prometheus.ome.svc:9090"

Add these values to the values file you install both charts with. Upgrade ome-crd first, then ome-resources, at the chart version you run:

helm upgrade --install ome-crd oci://ghcr.io/moirai-internal/charts/ome-crd \
  --namespace ome --version 1.3.0 -f values.yaml
helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that each release has been upgraded.

The ome-resources upgrade restarts the controller, which looks for the CRD and for KEDA only when it starts. A KEDA policy needs KEDA in the cluster: if you install KEDA later, or install the AutoscalerPolicy CRD without the chart, restart the controller yourself:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

Check that the CRD and the webhook are installed:

kubectl get crd autoscalerpolicies.ome.io -o name
kubectl get validatingwebhookconfiguration autoscalerpolicy.ome.io -o name

customresourcedefinition.apiextensions.k8s.io/autoscalerpolicies.ome.io
validatingwebhookconfiguration.admissionregistration.k8s.io/autoscalerpolicy.ome.io

Define a policy¶

This policy scales a component on the number of requests waiting in vLLM's queue, which the bundled Prometheus collects and labels with each pod's InferenceService and component:

apiVersion: ome.io/v1beta1
kind: AutoscalerPolicy
metadata:
  name: queue-depth
  namespace: llama-demo
spec:
  class: KEDA
  keda:
    triggers:
      - type: prometheus
        providerRef:
          name: cluster-prometheus
        metricType: AverageValue
        metadata:
          query: 'sum(vllm:num_requests_waiting{namespace="{{ .Namespace }}",inferenceservice="{{ .ISVCName }}",component="{{ .Component }}"})'
          threshold: "5"
          ignoreNullValues: "false"
    fallback:
      failureThreshold: 3
      replicas:
        fromComponent: MaxReplicas

Apply the file:

kubectl apply -f queue-depth.yaml

autoscalerpolicy.ome.io/queue-depth created

For each component that references it, OME renders this policy into a KEDA ScaledObject:

• The query sums the waiting requests across the component's pods. With metricType: AverageValue, KEDA divides the sum by the replica count, so it adds replicas to keep about 5 waiting requests per replica.
• ignoreNullValues: "false" makes an empty query result an error, instead of a zero.
• When the query fails more than 3 times in a row, KEDA scales the component to its maxReplicas.

The values in a trigger's metadata can use these variables, which OME fills in for each component:

A value holds literal text and these variables only: no template functions, pipelines or conditionals. The other fields:

OME rejects a policy that breaks these rules when you create or update it, including an invalid PromQL query and the one valid query shape that Queries that compute the replica count describes. AutoscalerPolicySpec in the API reference lists every field.

This HPA policy keeps CPU use at 60%, and when it scales down, it keeps the highest replica count recommended in the last 5 minutes:

apiVersion: ome.io/v1beta1
kind: AutoscalerPolicy
metadata:
  name: cpu-60
  namespace: llama-demo
spec:
  class: HPA
  hpa:
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            type: Utilization
            averageUtilization: 60
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300

kubectl apply -f cpu-60.yaml

autoscalerpolicy.ome.io/cpu-60 created

Queries that compute the replica count¶

In the queue-depth policy, the query measures load, and KEDA works out the replica count from the measurement. A query can instead compute the count itself. This policy scales a component to its maxReplicas while it has requests running, and back to its minReplicas when it has none:

apiVersion: ome.io/v1beta1
kind: AutoscalerPolicy
metadata:
  name: scale-to-max
  namespace: llama-demo
spec:
  class: KEDA
  keda:
    triggers:
      - type: prometheus
        providerRef:
          name: cluster-prometheus
        metricType: AverageValue
        queryReturnsDesiredReplicas: true
        metadata:
          query: '((sum(vllm:num_requests_running{namespace="{{ .Namespace }}",inferenceservice="{{ .ISVCName }}",component="{{ .Component }}"}) > bool 0) * {{ .MaxReplicas }})'
          threshold: "1"
          ignoreNullValues: "false"
    fallback:
      failureThreshold: 3
      replicas:
        fromComponent: MaxReplicas

Apply the file:

kubectl apply -f scale-to-max.yaml

autoscalerpolicy.ome.io/scale-to-max created

sum(...) > bool 0 is 1 while any request is running, otherwise 0, and * {{ .MaxReplicas }} turns that into the component's maxReplicas or 0. With metricType: AverageValue and threshold: "1", the HorizontalPodAutoscaler that KEDA creates divides the query's result by the threshold, rounds up, and clamps the count to the component's replica bounds. The query's result becomes the replica count, except that an idle component falls to its minReplicas, not to zero. As in the queue-depth policy, the fallback scales the component to its maxReplicas when the query keeps failing.

A query that computes the count needs queryReturnsDesiredReplicas: true and the parentheses around the comparison. The webhook rejects:

• queryReturnsDesiredReplicas: true on a trigger whose metricType isn't AverageValue, with the reason DesiredCountRequiresAverageValue. The field declares what the query returns, and changes nothing in the rendered trigger; the declaration matters because the other metric type breaks the count. With Value, the HorizontalPodAutoscaler also multiplies by the number of ready pods, which is right for a load metric, but turns a count of N into N times the pod count.
• The query without the parentheses around the comparison, with the reason PromQLInvalid. sum(...) > bool 0 * {{ .MaxReplicas }} is valid PromQL, but * binds tighter than >, so it means sum(...) > bool (0 * {{ .MaxReplicas }}) and returns 0 or 1 instead of 0 or the count. OME checks every prometheus trigger's query for this shape, a bool comparison with arithmetic on plain numbers on its right side, with or without queryReturnsDesiredReplicas. The rejection message quotes the comparison, rendered with sample values in place of the variables, and shows the shape of the fix: (sum(...) > bool N) * M.

Attach a policy to an InferenceService¶

Name the policy in a component's autoscalerPolicyRef. This version of isvc.yaml attaches queue-depth to the engine, and sets the engine's replica bounds:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-3-2-1b-instruct
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 4
    autoscalerPolicyRef:
      name: queue-depth

Apply the file:

kubectl apply -f isvc.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-3-2-1b-instruct configured

The engine still has OME's default HorizontalPodAutoscaler, and a default KEDA install rejects a ScaledObject for a workload that another HorizontalPodAutoscaler scales. Until you delete OME's, the InferenceService's ingress and status stop updating. Delete it, as Switch from HPA to KEDA explains:

kubectl delete hpa llama-3-2-1b-instruct-engine -n llama-demo

horizontalpodautoscaler.autoscaling "llama-3-2-1b-instruct-engine" deleted from llama-demo namespace

OME then creates the ScaledObject scaledobject-llama-3-2-1b-instruct-engine, and KEDA creates its own HorizontalPodAutoscaler for it. Read the rendered server address, query and fallback replica count:

kubectl get scaledobject scaledobject-llama-3-2-1b-instruct-engine -n llama-demo \
  -o jsonpath='{.spec.triggers[0].metadata.serverAddress}{"\n"}{.spec.triggers[0].metadata.query}{"\n"}{.spec.fallback.replicas}{"\n"}'

http://ome-prometheus.ome.svc:9090
sum(vllm:num_requests_waiting{namespace="llama-demo",inferenceservice="llama-3-2-1b-instruct",component="engine"})
4

The InferenceService reports where the engine's autoscaler came from:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='SOURCE:.status.components.engine.autoscaler.specSource,POLICY:.status.components.engine.autoscaler.policy.name,RESOLVED:.status.components.engine.autoscaler.conditions[?(@.type=="AutoscalerResolved")].reason'

SOURCE   POLICY        RESOLVED
policy   queue-depth   RenderedFromPolicy

The engine, decoder and router can each name a policy, the same one or different ones. The webhooks reject:

• any reference until the controller has found the CRD at startup, with a message that starts with AutoscalerPolicyFeatureDisabled;
• kind: ClusterAutoscalerPolicy in a reference, with AutoscalerPolicyRefKindReserved: spec.engine.autoscalerPolicyRef.kind "ClusterAutoscalerPolicy" is not supported (ClusterAutoscalerPolicy is a reserved shape); use AutoscalerPolicy. The schema lists the kind, but it's reserved until a cluster-scoped policy exists, so a reference needs only name: kind: AutoscalerPolicy is the default and the only kind there is;
• autoscalerPolicyRef in a runtime, because references go only in an InferenceService;
• minReplicas: 0 with only a reference: scale to zero needs the component's own autoscaler with class: KEDA.

Inline settings win¶

A component's inline autoscaler outranks its autoscalerPolicyRef, as Which setting wins lists. Set an inline autoscaler beside the reference to try other settings, to roll back, or to end a hold, without removing the reference. This engine keeps its reference but scales on CPU use:

spec:
  engine:
    minReplicas: 1
    maxReplicas: 4
    autoscalerPolicyRef:
      name: queue-depth
    autoscaler:
      class: HPA
      hpa:
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 70

OME replaces the ScaledObject with a HorizontalPodAutoscaler, and reports the reference as shadowed:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='SOURCE:.status.components.engine.autoscaler.specSource,SHADOWED:.status.components.engine.autoscaler.shadowedPolicyRef.name,RESOLVED:.status.components.engine.autoscaler.conditions[?(@.type=="AutoscalerResolved")].reason'

SOURCE   SHADOWED      RESOLVED
isvc     queue-depth   InlinePrecedence

OME still renders the shadowed policy on every reconcile, and reports the render in shadowedPolicyRef, so you can preview what the policy would dispatch while the inline settings keep running:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='SHADOWED:.status.components.engine.autoscaler.shadowedPolicyRef.name,PORTABLE:.status.components.engine.autoscaler.shadowedPolicyRef.portableDigest,WOULD-RENDER:.status.components.engine.autoscaler.shadowedPolicyRef.wouldRenderDigest'

SHADOWED      PORTABLE           WOULD-RENDER
queue-depth   pv1:9f2c41d07a8e   rv1:5b82d4c91f0a

portableDigest identifies the policy's spec, and wouldRenderDigest identifies the render the policy would produce for this component if you removed the inline block: Conditions and status describes both. Edit the shadowed policy, and the digests follow it. When the shadowed policy fails to render — it's missing, invalid or names an unbound provider — shadowedPolicyRef carries only the name, with no digests, and the component keeps running on the inline settings: an inline autoscaler wins even over a broken policy.

To go back to the policy, remove the autoscaler block, then delete OME's HorizontalPodAutoscaler again, as in Attach a policy to an InferenceService. The policy renders from the same inputs, so the component's policy.resolvedDigest equals the wouldRenderDigest you previewed, unless the policy, the component's replica bounds or the provider binding changed in between. Confirm that a policy edit landed shows how to read it.

Fail-closed behavior¶

When OME can't use a component's policy, the component holds: it keeps its last autoscaler, and never falls back to the runtime's settings or the default. OME holds in these cases:

While a component holds, OME keeps creating and updating its workload, and only the autoscaler is frozen. The component keeps the autoscaler that its policy last rendered; on a RawDeployment component, changes to minReplicas and maxReplicas still reach it. If the policy never rendered for the component, OME leaves its current autoscaler, if any, as it is.

The component's AutoscalerResolved condition is False with the hold's reason, and its message starts with holding last-known-good scaler:. When a hold starts, OME also records an AutoscalerPolicyHold Warning event on the InferenceService, such as engine autoscaler is holding last-known-good (PolicyNotFound): AutoscalerPolicy "queue-depth" not found in namespace llama-demo.

A held component recovers once you create or fix the policy, or bind its provider. If the CRD or KEDA was missing when the controller started, restart the controller after you install it.

Metric providers¶

A prometheus trigger names a metric provider in providerRef instead of a Prometheus address, so the same policy works in any cluster that binds the name. The cluster administrator binds provider names in the ome-resources value ome.metricProviders, as Turn on the feature does, and canary analysis uses the same bindings.

A binding's optional authSecretRef names the Secret name and key that hold a bearer token for the provider. This version of values.yaml adds one to cluster-prometheus:

ome:
  autoscalerPolicy:
    enabled: true
  metricProviders:
    cluster-prometheus:
      serverAddress: "http://ome-prometheus.ome.svc:9090"
      authSecretRef:
        name: prometheus-bearer
        key: token

Create the Secret in each namespace that uses the provider. OME renders the provider's triggers with authModes: bearer in their metadata and an authenticationRef named ome-metric-provider-<provider>, so the triggers for cluster-prometheus point at ome-metric-provider-cluster-prometheus. Before it creates or updates a component's ScaledObject, OME creates a KEDA TriggerAuthentication with that name in the InferenceService's namespace. The object points KEDA at the binding's Secret:

kubectl get triggerauthentication ome-metric-provider-cluster-prometheus -n llama-demo \
  -o jsonpath='{.spec.secretTargetRef[0].parameter}{"\n"}{.spec.secretTargetRef[0].name}{"\n"}{.spec.secretTargetRef[0].key}{"\n"}'

bearerToken
prometheus-bearer
token

OME never reads the Secret: KEDA resolves it through the TriggerAuthentication when it queries the provider, so the token never passes through OME, and never appears in the policy, the rendered trigger or the InferenceService's status. Don't edit or delete the object: OME restores it the next time it reconciles a component whose policy uses the provider. OME never deletes one, so the object stays in the namespace after the components that used the provider are gone.

Autoscaling triggers don't send the binding's headers. OME picks up a change to the bindings on its own.

Conditions and status¶

A policy's status says whether it's valid, how many components use it, and the digest that identifies its spec:

List the policies with their Ready reason:

kubectl get autoscalerpolicies -n llama-demo \
  -o custom-columns='NAME:.metadata.name,CLASS:.spec.class,READY:.status.conditions[?(@.type=="Ready")].status,REASON:.status.conditions[?(@.type=="Ready")].reason,ATTACHED:.status.attachedComponents'

NAME          CLASS   READY   REASON           ATTACHED
cpu-60        HPA     True    TemplatesValid   <none>
queue-depth   KEDA    True    TemplatesValid   1

ATTACHED is <none> while the policy is unused. kubectl ome get autoscalerpolicies shows the same status, with the digest in its wide output.

On the InferenceService, status.components.<component>.autoscaler.policy records the render that the component's live autoscaler came from, and shadowedPolicyRef describes a reference that an inline autoscaler outranks:

While a component holds, policy keeps the generation and digests of the last render that succeeded — the render the frozen autoscaler came from — so you can see which settings are standing. Read the result in status describes the autoscaler block's other fields.

OME sets AutoscalerResolved only on components with a reference:

Confirm that a policy edit landed¶

A policy edit reaches a component's autoscaler through a render, and the digests prove the render happened. Edit queue-depth — raise the trigger's threshold from "5" to "8", say — and apply it. The policy's status picks up the new spec:

kubectl get autoscalerpolicy queue-depth -n llama-demo \
  -o custom-columns='GENERATION:.metadata.generation,OBSERVED:.status.observedGeneration,DIGEST:.status.portableDigest'

GENERATION   OBSERVED   DIGEST
2            2          pv1:7d41c0b3a95e

OME re-renders every component that references the policy. Compare the engine's provenance with the policy:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='POLICY:.status.components.engine.autoscaler.policy.name,GENERATION:.status.components.engine.autoscaler.policy.observedGeneration,PORTABLE:.status.components.engine.autoscaler.policy.portableDigest,RESOLVED:.status.components.engine.autoscaler.policy.resolvedDigest'

POLICY        GENERATION   PORTABLE           RESOLVED
queue-depth   2            pv1:7d41c0b3a95e   rv1:3e95b1c84d06

The engine's observedGeneration and portableDigest match the policy's generation and digest: its live ScaledObject was rendered from the spec you just applied. The same check works across clusters: two clusters running the same policy spec report the same portableDigest, so you can compare a component's provenance in one cluster against the policy in another without reading the rendered objects.

When the digests don't match and the component's AutoscalerResolved condition is False, the component is holding: the provenance shows the render that's still standing, not the spec you applied. Fix what the condition's message names, and the digests catch up on the next render.

resolvedDigest tells renders apart rather than specs: it also changes when you change the component's minReplicas or maxReplicas, or the cluster's provider binding, with no policy edit at all. Use portableDigest to see that an edit landed, and resolvedDigest to see that a component's render changed — or, after you remove an inline block, that it matches the wouldRenderDigest previewed in Inline settings win.

Delete a policy¶

The webhook refuses to delete a policy that a component references. With the engine using queue-depth, as in Attach a policy to an InferenceService:

kubectl delete autoscalerpolicy queue-depth -n llama-demo

Error from server (Forbidden): admission webhook "autoscalerpolicy.ome-webhook-server.validator" denied the request: AutoscalerPolicy "queue-depth" is referenced by 1 component(s): llama-3-2-1b-instruct/engine; remove the refs first, or set the ome.io/allow-in-use-delete="true" annotation to force deletion

Remove the references first. A component whose reference you remove takes its settings from the next source in Which setting wins.

To delete a policy that components still reference, annotate it first:

kubectl annotate autoscalerpolicy queue-depth -n llama-demo ome.io/allow-in-use-delete=true
kubectl delete autoscalerpolicy queue-depth -n llama-demo

autoscalerpolicy.ome.io/queue-depth annotated
autoscalerpolicy.ome.io "queue-depth" deleted from llama-demo namespace

The components that referenced it then hold with the reason PolicyNotFound, and keep their last autoscaler:

kubectl get inferenceservice llama-3-2-1b-instruct -n llama-demo \
  -o custom-columns='SOURCE:.status.components.engine.autoscaler.specSource,POLICY:.status.components.engine.autoscaler.policy.name,RESOLVED:.status.components.engine.autoscaler.conditions[?(@.type=="AutoscalerResolved")].reason'

SOURCE   POLICY        RESOLVED
policy   queue-depth   PolicyNotFound

Next steps¶

• Component autoscaling: the autoscaler block that a policy renders into, and which setting wins.
• InferenceService: the components that reference a policy.
• kubectl ome autoscale: show the autoscaling in effect for each component, and which source supplied it.