Alfred configuration Unreleased: coming in v1.3

You configure Alfred through its Helm chart, ome-alfred, which you install separately from OME. Its policies and safety limits go under the chart's alfredConfig value, and alfred.ome.io/* annotations override some of them for one InferenceService.

Alfred is alpha. It starts in recommend-only mode, which reports the moves its policies would make and changes no workload. Execute mode, in which it also migrates OMENative Instances, is a further opt-in. For what the settings do, see Alfred and Alfred policies.

config.yaml keys¶

Set these keys under alfredConfig in your values file, and apply them with helm upgrade, as Tune a policy shows. The chart renders them into the ConfigMap alfred-config at each upgrade, so change your values, not the ConfigMap. Helm merges your maps into the chart's defaults key by key, and replaces lists whole. Because the chart has no values schema, Helm silently accepts a key outside alfredConfig, and Alfred ignores it.

Alfred parses the document strictly: an unknown key, a duplicate key or a value of the wrong type fails the load with parse config.yaml: .... schemaVersion is required, and a missing key takes its default. For an integer or duration key, 0 also means the default, so it can't turn a limit off, and a negative value fails. For a number or boolean key, 0 or false stays as set. Durations use Go syntax, such as 90s or 1h30m. Hot reload says what happens to a document that fails.

The simulation profiles, scheduling.profiles, are under Scheduling profiles.

Mode and loops¶

NodeConditionChange fires when any node condition appears, disappears, or changes its status or last transition time. NodeMaintenanceChange fires when a node starts or stops matching a maintenance trigger.

Defragmentation policy¶

Keys under policies.defragmentation:

Helm merges your sizePrior into the chart's, so a size you leave out keeps the chart's weight. When you shorten sizeLadder, set each size you drop to null, or Alfred rejects the document:

alfredConfig:
  policies:
    defragmentation:
      scoring:
        sizeLadder: [1, 2, 4]
        sizePrior:
          "1": 0.2
          "2": 0.3
          "4": 0.5
          "8": null

Node-health policy¶

Keys under policies.nodeHealth. Alfred reads node conditions that another agent sets, such as a GPU health checker.

A node is unhealthy while one of its trigger conditions is True, and Alfred plans to move its Instances off it. Suspect and unknown nodes aren't move targets. Health states defines the states.

Maintenance triggers¶

policies.nodeHealth.maintenance.triggers lists the node conditions, labels and taints that mark a node for planned maintenance. The list is empty by default, so add a trigger for each signal that your maintenance tooling sets. Each trigger has a name and exactly one of condition, label or taint:

A node is due for maintenance when it matches any trigger. Maintenance evacuations use the cooldowns under Workload defaults, not healthCooldownFloorMinutes.

alfredConfig:
  policies:
    nodeHealth:
      maintenance:
        triggers:
          - name: maintenance-pending
            condition:
              type: example.com/MaintenancePending
              status: "True"
          - name: patching-label
            label:
              key: maintenance.example.com/state
              value: patching
          - name: patching-taint
            taint:
              key: maintenance.example.com/patching
              effect: NoSchedule

Workload defaults¶

These keys apply to every InferenceService, and the workload annotations override some of them for one. Only defaultMovable and the defragmentation policy's use of perWorkloadCooldownMinutes apply in recommend-only mode. Cooldowns explains how the cooldowns combine.

Safety bounds¶

Maintenance windows¶

maintenanceWindows limits when Alfred plans defragmentation moves. Don't confuse it with maintenance triggers, which mark nodes for evacuation. Each window has these fields:

By default there are no windows, and Alfred may plan defragmentation at any time. Outside your windows, in both modes, it drops its planned defragmentation moves, except emergencies, and keeps its advice. Node-health and maintenance evacuations ignore windows. A window can't cross midnight, so split one that does into two.

alfredConfig:
  maintenanceWindows:
    - days: [Sat, Sun]
      start: "02:00"
      end: "06:00"

Spot nodes and tenants¶

Deployment modes¶

Alfred migrates only OMENative Instances. For RawDeployment and LeaderWorkerSet (deprecated) workloads, it records advice at most.

Alfred reads each component's mode from the InferenceService, never from its runtime. So a component that's OMENative only through its runtime gets RawDeploymentMigrationUnsupported advice. To let Alfred move it, declare the mode in the InferenceService, as An OMENative component shows RawDeploymentMigrationUnsupported shows.

Output and logging¶

Alfred logs JSON at info level. The chart has no value that changes its logging.

Hot reload¶

Every Alfred replica watches the ConfigMap and reloads config.yaml when it changes, without a restart. A new document applies from the next decision pass, and each replica logs config reloaded.

When a document fails to parse or validate, or the ConfigMap has no config.yaml key, each replica keeps its last good configuration. It logs config reload failed; keeping last-known-good and counts the failure in alfred_policy_reload_total{outcome="failure"}. It also records its own PolicyReloadFailed Warning event on the ConfigMap, which gives the first problem Alfred found. A configuration change doesn't take effect shows how to find these events. A replica also keeps its configuration if you delete the ConfigMap.

A replica that starts with a rejected document has no last good configuration, so it runs Alfred's built-in defaults, which are recommend-only. A helm upgrade that changes alfredConfig replaces every pod. So after an upgrade with a rejected alfredConfig, every replica runs the defaults, even if Alfred ran in execute mode before.

Workload annotations¶

Alfred reads these annotations from an InferenceService's metadata.annotations, and they override the defaults above for that InferenceService. The policies ignore an invalid value and use the default.

To stop Alfred from planning moves for an InferenceService:

kubectl annotate inferenceservice chat -n prod alfred.ome.io/movable=false

inferenceservice.ome.io/chat annotated

Alfred sees the change at its next observation and applies it from the next decision pass.

Helm chart values¶

The chart runs Alfred with leader election, using the Lease alfred.ome.io, and keeps Alfred's ConfigMaps and Lease in the release namespace. Alfred exits at startup when a duration is out of range or a simulation worker fails to load. It logs invalid startup configuration or unable to configure recommendation simulation with the error.

Migration¶

migration.apiVersion: v1 lets Alfred plan moves for OMENative Instances, in either mode:

• By default the value is empty, and OMENative Instances get advice that names no target nodes, usually with the reason OMENativeUnavailable. This is expected, and Alfred metrics and events lists the metric and event that also show it.
• With it, each planned move names up to 3 target nodes in hintTargets. In recommend-only mode, the move stays advice, with the reason SimulationRecommendOnly, or a scheduling reason when no profile fits. In execute mode, Alfred may request it.

The value needs simulation.configMapName, or the chart fails with migration.apiVersion requires simulation.configMapName. Any value but v1 fails with migration.apiVersion must be v1 when enabled. It also renders:

• The migration guard, the ValidatingAdmissionPolicy and binding ome-alfred-migration-writes. It fails closed, and lets Alfred's ServiceAccount add only one bounded ome.io/migration-request-v1-<uuid> annotation, the same request as Request an instance migration. Its denials start with Alfred may. It needs Kubernetes 1.30 or later: on an older cluster, the install or upgrade fails.
• The journal, the ConfigMap alfred-dispatch-state, which Helm keeps when you uninstall the chart. Never delete or reset it. If it exists without state.json, the chart fails with existing alfred-dispatch-state is missing state.json; restore its journal before enabling migration.

A request that isn't acknowledged within migration.acknowledgementTimeout becomes stalled. Alfred never cancels it, and it blocks every later request. Alfred's first request stalls this way when the OME manager's InferenceReplica controller is off, for example with --enable-inferencereplica-controller=false. For the whole setup, see Let Alfred migrate Instances.

Simulation¶

Simulation predicts whether your scheduler would place a move's replacement pods. In recommend-only mode, Alfred records the prediction. In execute mode, it requests only moves that the simulation places.

simulation.configMapName names a ConfigMap that you create, with workers.json and a scheduler configuration file for each worker. The chart mounts it at /etc/alfred-simulation. Set up scheduler simulation for Alfred builds one.

Make the ConfigMap immutable: Alfred loads it once, at startup. To change it, create a new ConfigMap and set simulation.configMapName to its name, which rolls the pods. A worker's identity includes a hash of Alfred's build, so a new Alfred image needs a new ConfigMap and new profiles.

Alfred exits at startup when a worker reports an identity other than the one in workers.json, or when a worker's scheduler configuration file breaks one of these rules:

• It's a kubescheduler.config.k8s.io/v1 KubeSchedulerConfiguration that passes Kubernetes' validation.
• It has exactly one profile.
• It sets no clientConnection.kubeconfig and no extenders.
• Its enabled plugins and its pluginConfig name only Kubernetes' in-tree plugins and OMEGangPack. The upstream GangScheduling plugin isn't supported.
• If OMEGangPack runs, it runs at all of PreFilter, Filter, PostFilter, Reserve, Permit and PostBind, as enabling it under multiPoint does. It also needs pluginConfig args with a positive gcIntervalSeconds.

The OME scheduler's configuration, as its chart renders it, meets these rules. The simulator runs the scheduler of Kubernetes v1.35.4, whatever version your cluster runs.

Simulation needs the PodGroup CRD (scheduling.x-k8s.io/v1alpha1), which no chart installs. Without it, simulations fail with CaptureFailed, and execute mode withholds moves with SnapshotUnavailable. See Gang scheduling.

Scheduling profiles¶

The config.yaml key scheduling.profiles maps each scheduler name that replacement pods use to the simulation worker that models that scheduler. A pod without a schedulerName uses default-scheduler. Alfred selects only the profile with the pods' scheduler name, with no fallback.

Copy the values from what the worker prints with --print-profile, as Set up scheduler simulation for Alfred shows. A profile that matches no loaded worker makes each of its simulations fail. In recommend-only mode, the recommendation's scheduling.reason is WorkerFailed. In execute mode, Alfred withholds the move with SimulationNotFeasible.

This example has profiles for the default scheduler and for the OME scheduler, which models gangs:

alfredConfig:
  scheduling:
    profiles:
      default-scheduler:
        backend: kube-v135
        schedulerVersion: v1.35.4
        configurationID: sha256:<default-id>
      ome-scheduler:
        backend: ome-v135
        schedulerVersion: v1.35.4
        configurationID: sha256:<ome-id>
        gangScheduling: true

Alfred adds a scheduling object to each OMENative recommendation that names an Instance, with the profile it selected, if any, and a status and reason. Scheduling results lists the reasons. In execute mode with migration.apiVersion: v1, recommendations have no scheduling object: Alfred simulates each move just before it requests it. It withholds the move with SourceUnsupported when no profile fits, or with SimulationNotFeasible when the simulation fails or returns a status other than Feasible.

Related pages¶

• Alfred: what Alfred is and how its loops, modes and outputs fit together.
• Alfred policies: how the defragmentation and node-health policies choose moves.
• Run Alfred in recommend-only mode: install the chart and read its recommendations.
• Let Alfred migrate Instances: turn on execute mode.
• Alfred metrics and events: what Alfred exports and records.