Alfred policies Unreleased: coming in v1.3

Alfred runs two policies in its decision passes, 5 minutes apart by default. Defragmentation looks for moves that gather scattered free GPUs into blocks that a large Instance can use. Node health moves Instances off nodes whose GPUs have failed or that are due for planned work. Both are on by default. A move that Alfred can't make, or won't make in recommend-only mode, is advice with a reason.

Alfred is alpha, and it only moves OMENative Instances. It starts in recommend-only mode, where it changes no workload. In execute mode, a further opt-in that Let Alfred migrate Instances sets up, the arbiter picks which moves go ahead, within the safety bounds. You set the policies under alfredConfig in the ome-alfred chart's values: see Alfred configuration.

Which Instances Alfred can move¶

Alfred reads a component's deployment mode from the InferenceService alone. RawDeployment Instances are always advice, with the reason RawDeploymentMigrationUnsupported. A component that's OMENative only through its serving runtime counts as a RawDeployment, so declare OMENative in the InferenceService. A MultiNode component gets LWSMigrationUnsupported.

An OMENative Instance can move when all of these hold. Otherwise its move is advice, with the reason shown:

OMENative moves an Instance by surge: it starts the new pods first, and removes the old ones once the new ones serve. See Migration and transient scale.

Defragmentation¶

A GPU cluster fragments as Instances come and go. Free GPUs end up a few to a node, so an Instance that needs a whole 8-GPU node stays Pending even when the cluster has more than 8 free. Moving a 2-GPU Instance off a half-empty node onto a fuller one can free the whole node. Defragmentation measures this damage and looks for such moves. policies.defragmentation.enabled: false turns it off.

Alfred scores hardware pools separately, since only nodes with the same kind of GPU can stand in for one another. Instances without GPUs stay put.

How Alfred scores a pool¶

Alfred counts a pool's free GPUs on the nodes that could take a new pod. That leaves out nodes that are cordoned, quarantined by node health, due for maintenance or marked for scale-down by the cluster autoscaler. By default, it also leaves out preemptible nodes.

The score has two parts, and either can wake the policy alone:

• Reclaimable fragmentation: free GPUs that the Instance sizes the pool needs can't use now, but could once Alfred repacks the Instances it can move.
• Pending pressure: Pending GPU pods that fit nowhere now but would after that repack, weighted by how long they've waited.

fragmentationThreshold, 0.25 by default, decides what happens next:

• When the score is above it, Alfred plans moves, as How Alfred picks a move describes.
• When only the fragmentation that no move can fix is above it, the pool's OMENative Instances are advice with NonExecutableObservedFragmentation.
• Otherwise, the pool gets no recommendations.

Until you set migration.apiVersion, no Instance can move, so the score stays at 0 and only the second case can happen.

slots(s)         = number of size-s Instances the pool's free GPUs can seat
frag(s)          = 1 - slots(s) * s / free GPUs
weight(s)        = (1 - lambda) * demand share(s) + lambda * prior(s)
observed         = sum over the ladder of weight(s) * frag(s)
best             = observed, recomputed after repacking the movable Instances
reclaimable      = max(0, observed - best)
urgency          = 1 - exp(-minutes pending / tau)
pending pressure = 1 - product over repackable pending pods of (1 - urgency)
score            = 1 - (1 - reclaimable) * (1 - pending pressure)

How Alfred picks a move¶

When a pool scores above the threshold, Alfred plans surge moves for the pool's OMENative Instances that can move:

1. It picks target nodes. Alfred tries the fullest nodes with room first. They must take new pods and have access to the model.
2. It weighs benefit against cost. The move must lower the pool's fragmentation by more than its cost. The cost depends on policies.defragmentation.aggressiveness, which changes nothing else: 0.225 for conservative, 0.15 for balanced, the default, and 0.075 for aggressive. A move that falls short is advice with NonExecutableObservedFragmentation.
3. It boosts urgent moves. A move scores double, and counts as an emergency, when it would seat a Pending pod that fits nowhere else and has waited longer than emergencyPendingAgeMinutes, 10 by default. The pod must be in the moved InferenceService's namespace, or belong to an InferenceService with the same alfred.ome.io/tenant-group. allowCrossTenantOptimization: false limits it to the namespace. By default, a move off a preemptible node scores 1.25 times higher.

Apart from the emergency boost, defragmentation ignores namespaces: any Instance can make room for any other's. A move names up to 3 target nodes as hints, but the scheduler places the new pods, so execute mode simulates the move first.

Node health¶

The node-health policy moves Instances off nodes whose GPUs have failed, and off nodes you've marked for planned work. policies.nodeHealth.enabled: false turns it off. By default, a change to a node's conditions or maintenance triggers starts an extra decision pass (earlyTickOn).

An Instance with a pod on an Unhealthy node gets a move with the reason NodeUnhealthy, and one on a node due for maintenance gets NodeMaintenance. Alfred plans the same surge move as for defragmentation, but skips the threshold and the cost, and ignores maintenance windows. The arbiter considers these moves before defragmentation's.

Health states¶

Alfred reads a node's health from the node conditions in policies.nodeHealth.triggerConditions, only GpuUnhealthy by default. OME doesn't set that condition: your GPU health checks must set it, or you name the conditions they do set. When conditions disagree, the worst state wins:

Planned maintenance¶

policies.nodeHealth.maintenance.triggers marks nodes due for planned work, and is empty by default. A trigger has a unique name and matches nodes by one condition, label or taint:

alfredConfig:
  policies:
    nodeHealth:
      maintenance:
        triggers:
          - name: os-patching
            label:
              key: maintenance.example.com/state
              value: patching
          - name: hardware-swap
            taint:
              key: maintenance.example.com/hardware-swap
              effect: NoSchedule

Alfred places nothing on a node that matches a trigger. Moves off such a node wait out the full cooldowns, unlike moves off Unhealthy nodes. Maintenance triggers lists the fields.

Signals and events¶

Alfred lists the nodes that aren't Clear, or are due for maintenance, in its recommendations, under node.<name> keys. It also emits events on the Node:

With signalOnly: true, the policy reports nodes and moves nothing. Use it when your own repair tooling drains nodes and acts on these events. Alfred metrics and events describes the events.

Simulation¶

Alfred's plan counts GPUs, but the scheduler also weighs taints, affinity, topology spread and gangs. So Alfred asks a simulator where the new pods would land. The simulator runs the scheduler's code against a copy of the cluster. You set it up with the chart's simulation.configMapName, as Set up scheduler simulation for Alfred shows.

Alfred simulates with the scheduler profile that the Instance's schedulerName names, or default-scheduler when it's empty. An Instance with more than one pod needs a profile that simulates gangs, like the one for the OME scheduler, which is alpha and needs Kubernetes 1.35. When no profile fits, the recommendation names the problem, such as GangUnsupported, and execute mode withholds the move with SourceUnsupported.

Simulation also needs the PodGroup CRD, even when no pod uses PodGroups: see Gang scheduling.

In recommend-only mode¶

Simulation works without migration.apiVersion. Alfred simulates up to 8 moves a pass, and records their results, Feasible, Infeasible or Unsupported, with the predicted pod placements. A prediction reserves no capacity.

Once you set migration.apiVersion, the policies plan moves and their target nodes, and Alfred makes the planned moves advice with SimulationRecommendOnly.

In execute mode¶

In execute mode, just before Alfred submits a move, it looks at the cluster afresh, reruns the policies and the arbiter, and simulates the move, which must come out Feasible. It then checks that nothing the decision relied on has changed, and that the pods land on allowed, ready nodes outside their cooldown. Otherwise it withholds the request, with a reason such as PredictedTargetUnsafe. That's why migration.apiVersion needs simulation.

The arbiter¶

In recommend-only mode, new moves are advice before they reach the arbiter, so their outcome is advisory. Requests left in Alfred's dispatch journal from execute mode keep their status, and a prepared one shows as withheld with ExecutionDisabled.

In execute mode, the arbiter considers the moves from both policies one at a time, in this order:

1. By class: moves off Unhealthy nodes, then off nodes due for maintenance, then defragmentation moves.
2. By score, highest first. A node-health move scores its alfred.ome.io/priority annotation, 0.5 by default. A defragmentation move scores its benefit minus cost, times any boost.
3. By size, smaller Instances first.

It admits a move that passes its checks: the global limits and cooldowns, one move per InferenceService and per target node in a pass, and room on a target node. The first check a move fails gives its rejection reason, and Arbiter reasons lists them all.

Admission isn't submission. Alfred requests the first admitted move that passes the checks in In execute mode, and withholds the rest with the reasons in Dispatch reasons.

Safety bounds¶

Maintenance windows, the spot rules and defragmentation's per-workload cooldown act inside the policies, so they shape recommendations in either mode. The global limits and the other cooldowns act in the arbiter and when Alfred submits a request, so they only hold back execute mode.

Global limits¶

• Caps: maxInFlightMigrations, 3 by default, bounds the migrations in progress, and maxMigrationsPerHour, 10 by default, those started in the last hour. Both count everyone's migrations, not only Alfred's.
• One at a time: Alfred submits at most one request a pass, and none while any migration request in the cluster is waiting or in progress (InFlightCap).
• Unresolved requests: Alfred submits nothing while one of its own requests is unresolved (UnresolvedRequest). A request that OMENative doesn't pick up within migration.acknowledgementTimeout, 2 minutes by default, is marked stalled, but still counts until OMENative reports its end.
• Failure backoff: after one of Alfred's migrations fails or stalls, Alfred waits migration.failureBackoff, 5 minutes by default, before it submits another (FailureBackoff).
• Circuit breaker: when more than half of Alfred's recent migrations failed, it admits nothing for 60 minutes (CircuitBreakerOpen). A new leader starts the count afresh.

Cooldowns¶

Maintenance windows¶

maintenanceWindows limits when defragmentation may move anything. A window lists days, from Mon to Sun, and a start and an end time in UTC. The start must come before the end, so a window can't cross midnight:

alfredConfig:
  maintenanceWindows:
    - days: [Sat, Sun]
      start: "01:00"
      end: "05:00"

With no windows, the default, defragmentation can act at any time. Outside the windows, it keeps its advice but drops its moves, except emergencies.

Spot and preemptible nodes¶

Alfred treats a node as preemptible when it has one of the labels in spotPolicy.preemptibleLabels, with any value other than false. By default, those are node.kubernetes.io/preemptible and cloud.google.com/gke-preemptible. Alfred keeps moves off such nodes and leaves their free GPUs out of the score (spotPolicy.avoidAsTarget), and boosts defragmentation moves off them (spotPolicy.preferAsSource). Both are on by default. The alfred.ome.io/spot-policy annotation adjusts this for one InferenceService. Its value avoid keeps its moves off preemptible nodes, even when avoidAsTarget is false. With migrate, Alfred always boosts its moves off them, and with ignore, never.

Per-workload annotations¶

You set Alfred's annotations on an InferenceService's own metadata.annotations. alfred.ome.io/movable: "false" opts it out, and "true" opts it in when defaultMovable is false. The other alfred.ome.io/ annotations, priority, cooldown-minutes, tenant-group and spot-policy, tune its moves. When it plans, Alfred ignores a value it can't parse and uses the default. Workload annotations lists their values.

Next steps¶

• Run Alfred in recommend-only mode: install the chart, read what the policies recommend and tune a policy.
• Set up scheduler simulation for Alfred: predict where the replacement pods of each move would land.
• Let Alfred migrate Instances: turn on execute mode.
• Alfred configuration: the settings and chart values, with their defaults.
• Alfred metrics and events: the reasons, metrics and events the policies produce.