Scale to zero with KEDA Unreleased: coming in v1.3

Scale an idle InferenceService component to zero replicas, so that it holds no GPUs, and bring it back when a KEDA trigger becomes active. Give the component its own autoscaler with class: KEDA and at least one trigger, and OME creates a KEDA ScaledObject that scales the component's Deployment. This works for components in the RawDeployment mode, the default for a component without a leader and workers: see Deployment modes. OMENative components keep at least one Instance. The autoscaler field is alpha, so its API can change between releases. It needs no feature flag.

What happens at zero¶

While a component is at zero, no pod serves it, and requests to it fail. Requests don't wake it up: only a KEDA trigger does. The component then serves again once a new pod is ready, so plan for the time that the engine takes to start and load the model.

Pick triggers whose source keeps reporting while the component is at zero, such as a schedule, a queue, or a metric from a service that stays up. The component's own metrics stop with its pods, so they can't wake it. For a metric in the Prometheus that the ome-resources chart deploys, point a prometheus trigger at http://ome-prometheus.ome.svc:9090, for OME in the namespace ome. See Collect metrics.

Step 1: Create an InferenceService that scales to zero¶

Create a namespace for the example:

kubectl create namespace llama-demo

namespace/llama-demo created

Save this InferenceService as llama-chat.yaml. Its engine scales between zero and two replicas, and a KEDA cron trigger asks for one replica from 09:00 to 17:00 UTC each day:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 0
    maxReplicas: 2
    autoscaler:
      class: KEDA
      keda:
        triggers:
          - type: cron
            metadata:
              timezone: UTC
              start: "0 9 * * *"
              end: "0 17 * * *"
              desiredReplicas: "1"

If it's between 09:00 and 17:00 UTC now, move start and end to earlier hours today, so that the engine scales to zero in Step 3.

OME copies the keda block into the ScaledObject as written, so you can use any trigger that your KEDA version supports: see the ScaledObject specification. The block's optional fields include these:

Apply the file:

kubectl apply -f llama-chat.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-chat created

Step 2: Check the ScaledObject¶

OME creates the Deployment llama-chat-engine and the ScaledObject scaledobject-llama-chat-engine. From then on, KEDA sets the Deployment's replica count. Check that the ScaledObject scales the Deployment between the engine's minReplicas and maxReplicas:

kubectl get scaledobject scaledobject-llama-chat-engine -n llama-demo -o jsonpath='{.spec.scaleTargetRef.kind}/{.spec.scaleTargetRef.name} {.spec.minReplicaCount}-{.spec.maxReplicaCount}{"\n"}'

Deployment/llama-chat-engine 0-2

If the output names an InferenceReplica instead, the engine runs on OMENative, which keeps at least one Instance.

Step 3: Watch the engine scale to zero¶

Outside the trigger's window, KEDA scales a new engine to zero the first time it checks the trigger. Later, when the window closes, KEDA waits for cooldownPeriod before it scales the engine to zero. Check the Deployment's replica count:

kubectl get deployment llama-chat-engine -n llama-demo -o jsonpath='{.spec.replicas}{"\n"}'

0

At zero, the InferenceService reports Ready, but its model status is pending:

kubectl get inferenceservice llama-chat -n llama-demo -o jsonpath='{.status.modelStatus.transitionStatus} {.status.modelStatus.modelRevisionStates.targetModelState}{"\n"}'

InProgress Pending

When the window opens, KEDA scales the Deployment to one replica, and the engine serves again once its pod is ready.

Keep a minimum while a trigger is active¶

To idle at zero but keep a minimum while a trigger is active, set keda.idleReplicaCount to 0 and minReplicas to that minimum. Change llama-chat.yaml to this:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 2
    maxReplicas: 4
    autoscaler:
      class: KEDA
      keda:
        idleReplicaCount: 0
        triggers:
          - type: cron
            metadata:
              timezone: UTC
              start: "0 9 * * *"
              end: "0 17 * * *"
              desiredReplicas: "3"

Apply the file:

kubectl apply -f llama-chat.yaml

Warning: Runtime vllm-llama-3-2-1b-instruct will be auto-selected for model llama-3-2-1b-instruct
inferenceservice.ome.io/llama-chat configured

Check the ScaledObject's counts:

kubectl get scaledobject scaledobject-llama-chat-engine -n llama-demo -o jsonpath='{.spec.minReplicaCount}-{.spec.maxReplicaCount} {.spec.idleReplicaCount}{"\n"}'

2-4 0

While no trigger is active, KEDA scales the engine to zero. While the trigger is active, it scales the engine between 2 and 4 replicas. idleReplicaCount must be lower than minReplicas, so leave it out when minReplicas is 0.

OMENative components can't scale to zero¶

An OMENative component keeps at least one Instance. With minReplicas: 0, the webhook admits the InferenceService, but OME sets the ScaledObject's minReplicaCount to 1, and kubectl ome autoscale explain shows unsupported in the ZERO row. With keda.idleReplicaCount: 0, explain shows yes in the ZERO row, but while no trigger is active the component drops to one Instance. OME then sets the count back to minReplicas, so the component can grow back to minReplicas Instances.

A component with a leader and workers can't scale to zero either: it runs on OMENative, or in MultiNode (deprecated), which gets no autoscaler.

To scale to zero, run the component as a RawDeployment. The InferenceService's spec.deploymentMode, or an ome.io/deploymentMode annotation on the InferenceService or its runtime, can make a component OMENative: see How OME resolves the mode.

Troubleshooting¶

The webhook rejects the InferenceService¶

kubectl apply fails with a message that contains one of these reasons:

The InvalidScaleToZero message also suggests the ome.io/autoscalerClass=keda annotation. Use the component's own autoscaler instead: OME needs its triggers to create the ScaledObject.

No ScaledObject appears¶

The webhook admits the InferenceService, but OME fails to create the ScaledObject, so the component never scales to zero. OME records no event, and leaves the InferenceService's status and ingress as they were. Read the logs of every manager replica:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

Each failed attempt logs Failed to reconcile component with an error:

To restart the controller after you install KEDA, so that it watches ScaledObjects:

kubectl rollout restart deployment/ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

To delete OME's HorizontalPodAutoscaler after a switch in one step:

kubectl delete hpa llama-chat-engine -n llama-demo

horizontalpodautoscaler.autoscaling "llama-chat-engine" deleted from llama-demo namespace

With class: KEDA, OME doesn't create it again, and it creates the ScaledObject when it retries.

The component doesn't wake up¶

KEDA scales the component up only when a trigger becomes active, and a trigger on the component's own metrics has nothing to read at zero. Use a trigger whose source keeps reporting at zero, as What happens at zero describes. The ScaledObject's Active condition, in the InferenceService's status.components.engine.autoscaler.conditions, shows whether a trigger is active.

Clean up¶

Delete the InferenceService and its namespace:

kubectl delete inferenceservice llama-chat -n llama-demo
kubectl delete namespace llama-demo

inferenceservice.ome.io "llama-chat" deleted from llama-demo namespace
namespace "llama-demo" deleted

Deleting the InferenceService also deletes its ScaledObject.

Next steps¶

• Component autoscaling: how OME picks and runs each component's autoscaler.
• Autoscaler policy: share KEDA or HPA settings across InferenceServices with an alpha AutoscalerPolicy.
• Bring your own autoscaler: scale a component with an autoscaler that you run yourself.
• kubectl ome autoscale: see the autoscaling in effect for each component, and where it came from.