Bring your own autoscaler Unreleased: coming in v1.3

Set a component's autoscaler.class to External to scale it with an autoscaler that you run: a HorizontalPodAutoscaler or KEDA ScaledObject that you write, or a controller of your own. OME then creates no autoscaler for the component, and publishes the object for yours to scale in status.components.<component>.scaleTargetRef. The autoscaler field and the autoscaler and scaleTargetRef status fields are alpha, so they can change between releases, but they need no feature flag. On v1.2.2, the ome.io/autoscalerClass: external annotation stops OME from creating a HorizontalPodAutoscaler for a RawDeployment InferenceService.

Step 1: Declare the class¶

Set class: External in the component's autoscaler block. Save this InferenceService as chat.yaml:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat
  namespace: prod
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-3b-instruct
  runtime:
    name: srt-llama-3-2-3b-instruct
  engine:
    minReplicas: 2
    autoscaler:
      class: External

External needs no other settings. OME creates the InferenceReplica with the engine's minReplicas, 2 Instances here, and then your autoscaler sets the count. A RawDeployment component starts with 1 pod, whatever its minReplicas. On a running component, OME deletes the autoscaler that it created, and the count stays put until yours changes it. Add the same block under decoder or router for any other component that your autoscaler scales.

To hand off the engine of all the InferenceServices that use a runtime and set no autoscaling of their own, put the block in the runtime's engineConfig instead. To hold a component at a fixed count, use class: None.

Apply the file:

kubectl apply -f chat.yaml

inferenceservice.ome.io/chat created

Check the class that OME applied to the engine:

kubectl get inferenceservice chat -n prod -o jsonpath='{.status.components.engine.autoscaler}'

{"class":"External","managedBy":"external","specSource":"isvc"}

specSource: isvc says that the setting came from the InferenceService. An inline autoscaler block wins over all other autoscaling settings, as Which setting wins shows.

Step 2: Read the published scale target¶

Read the object to scale from the engine's status:

kubectl get inferenceservice chat -n prod -o jsonpath='{.status.components.engine.scaleTargetRef}'

{"apiVersion":"ome.io/v1beta1","kind":"InferenceReplica","name":"chat-engine"}

The target depends on the component's deployment mode:

Your autoscaler writes the count to the target's scale subresource, as kubectl scale does, and OME adds or removes Instances to match. Write the count only that way: OME owns the rest of the InferenceReplica's spec and overwrites other changes to it.

Step 3: Point your autoscaler at the target¶

Whichever autoscaler you run:

• Point one autoscaler at each target. Two overwrite each other's counts.
• Set the bounds in your autoscaler. OME passes minReplicas and maxReplicas only to the autoscalers that it creates.
• On OMENative, keep your autoscaler's minimum at 1 or more. OME sets a count of 0 back to minReplicas, or to 1 when minReplicas is unset or 0.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: chat-engine-byo
  namespace: prod
spec:
  scaleTargetRef:
    apiVersion: ome.io/v1beta1
    kind: InferenceReplica
    name: chat-engine
  minReplicas: 3
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

kubectl apply -f chat-engine-byo.yaml

horizontalpodautoscaler.autoscaling/chat-engine-byo created

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: chat-scaler
  namespace: prod
rules:
  - apiGroups: ["ome.io"]
    resources: ["inferenceservices"]
    verbs: ["get"]
  - apiGroups: ["ome.io"]
    resources: ["inferencereplicas/scale"]
    verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: chat-scaler
  namespace: prod
subjects:
  - kind: ServiceAccount
    name: chat-scaler
    namespace: prod
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: chat-scaler

kubectl apply -f chat-scaler-rbac.yaml

role.rbac.authorization.k8s.io/chat-scaler created
rolebinding.rbac.authorization.k8s.io/chat-scaler created

Step 4: Check the scaling¶

List the InferenceReplica, whose short name is irep:

kubectl get inferencereplica chat-engine -n prod

NAME          COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
chat-engine   engine      3         3         2       2           25m

DESIRED is the count that your autoscaler last wrote: here, the HorizontalPodAutoscaler's minimum of 3, not the engine's minReplicas of 2. CURRENT counts the Instances that exist, and READY and AVAILABLE catch up as the new Instance starts. For a RawDeployment component, kubectl get deployment chat-engine -n prod shows the same in pods. OME doesn't copy your autoscaler's decisions into its status, so read them from your autoscaler.

Hand scaling back to OME¶

To hand a component back to OME, delete your autoscaler first. Then set another class, or remove the autoscaler block so that the next source in Which setting wins applies.

Troubleshooting¶

Your count doesn't last¶

Something other than your autoscaler sets the count. Read the class that OME applied, which points to the cause:

kubectl get inferenceservice chat -n prod -o jsonpath='{.status.components.engine.autoscaler.class}'

• None: when minReplicas is 1 or more, as it is by default, OME sets the count back to it. Set class: External.
• HPA or KEDA: OME's own autoscaler scales the component too. Put the autoscaler block under the component that your autoscaler targets. specSource in the same status says where the class came from.
• External: a second autoscaler targets the same object, or yours writes 0 to an OMENative component. Keep one autoscaler per target, with a minimum of 1 or more.

scaleTargetRef is empty¶

The Step 2 command prints nothing for a MultiNode (deprecated) component, which has no scale target: move it to OMENative. Otherwise, OME hasn't set up the component yet, or setting it up failed. Look for a Reconciler error line in the controller logs:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

Fix the error that it names, and OME retries on its own.

Switching back to HPA or KEDA fails¶

A switch to class: HPA or class: KEDA fails when an object of yours has the name that OME uses for its own: OME won't take it over. For this engine, those names are chat-engine for a HorizontalPodAutoscaler and scaledobject-chat-engine for a ScaledObject. Each attempt logs a Reconciler error line that names the object and says it is not controlled by expected owner. Delete or rename your object, and OME creates its own on the next attempt.

Clean up¶

kubectl delete -f chat-engine-byo.yaml -f chat.yaml

horizontalpodautoscaler.autoscaling "chat-engine-byo" deleted from prod namespace
inferenceservice.ome.io "chat" deleted from prod namespace

kubectl delete -f chat.yaml

inferenceservice.ome.io "chat" deleted from prod namespace

kubectl delete -f chat-scaler-rbac.yaml -f chat.yaml

role.rbac.authorization.k8s.io "chat-scaler" deleted from prod namespace
rolebinding.rbac.authorization.k8s.io "chat-scaler" deleted from prod namespace
inferenceservice.ome.io "chat" deleted from prod namespace

Next steps¶

• Component autoscaling: every autoscaler class, including None for a fixed count, and how OME picks a component's settings.
• Scale to zero with KEDA: let OME's KEDA ScaledObject scale a component to zero.
• Autoscaler policy: share HPA or KEDA settings across InferenceServices. It's alpha and off by default.
• Request a transient scale: ask once for a different count with kubectl ome scale, which is alpha.
• kubectl ome autoscale: see the autoscaling in effect for each component, and where it came from.