Alfred metrics and events Unreleased: coming in v1.3

Alfred reports what it observes and recommends through Prometheus metrics, Kubernetes events and the ConfigMap alfred-recommendations. With migration.apiVersion: v1, the ConfigMap alfred-dispatch-state also tracks its migration requests. Alfred is alpha. It runs only when you install its opt-in chart, ome-alfred, and it starts in recommend-only mode, which changes no workload. Execute mode, in which Alfred migrates OMENative Instances, is a further opt-in. For what Alfred does, see Alfred. To install it, see Run Alfred in recommend-only mode.

The chart runs 3 replicas. Every replica observes the cluster every 30 seconds (observationLoopInterval) and publishes the cluster metrics. The leader, the replica that holds the Lease alfred.ome.io, also runs a decision pass every 5 minutes (decisionLoopInterval), and early on the node changes in earlyTickOn. It publishes each pass as decision metrics, events and the recommendations ConfigMap. See High availability.

The chart values are configMapName and the settings named migration.*, simulation.* and metrics.*. The others are keys of Alfred's config.yaml, which you set through the chart's alfredConfig values. Alfred configuration lists both, with their defaults.

In the default install, migration.apiVersion is empty, so Alfred has no migration executor. That's expected, and it shows in these ways:

• alfred_omenative_unavailable is 1 on every replica.
• Each process that becomes leader records one OMENativeUnavailable Warning event.
• Each steady OMENative Instance that a policy picks gets OMENativeUnavailable advice, with no target hints.
• The fragmentation score, reclaimable fragmentation and pending pressure stay at 0, because no Instance counts as movable.

The defragmentation policy still gives advice when a pool's demand-weighted fragmentation is above fragmentationThreshold, a figure no metric shows. See How Alfred scores a pool. Planned moves, with target hints, need migration.apiVersion: v1, even in recommend-only mode. See Let Alfred migrate Instances.

Metrics¶

Each replica serves /metrics on port 8080, the container port metrics (the chart value metrics.port). The chart annotates each pod with prometheus.io/scrape, prometheus.io/port and prometheus.io/path. With metrics.serviceMonitor.enabled: true, it also creates the ServiceMonitor ome-alfred, which needs the Prometheus Operator CRDs.

Scrape every pod, as the ServiceMonitor and the annotations do. The Service ome-alfred-metrics sends each request to one replica, and only the leader publishes decision metrics. Counters start at 0 when a pod starts. The endpoint also serves the Go runtime and controller-runtime metrics. To read the leader's metrics by hand, see Step 3: Check its metrics.

Cluster metrics¶

Every replica republishes these gauges on each observation, and a pool, node or size that's gone drops out. A node's pool is its nvidia.com/gpu.product label, or else its instance type, shortened for known shapes, or else its GPU resource name.

The fragmentation, pressure and surge headroom figures leave out nodes that can't take work. Those are nodes whose health state isn't Clear, and nodes that have maintenance requested, are cordoned or are marked for scale-down by the cluster autoscaler. With spotPolicy.avoidAsTarget, the default, the fragmentation and pressure figures also leave out spot nodes.

Decision metrics¶

The leader updates these counters after each decision pass. The recommendation counters count every row, so a row that persists counts again in each pass. The signal counters count node events. workload is the InferenceService as namespace/name. policy is defragmentation or nodehealth, or migration-dispatch for a journal entry whose request Alfred can't read.

To follow migration requests and their results, read the recommendations ConfigMap and the events.

Loop and state metrics¶

Both histograms use the default Prometheus buckets, 0.005 to 10 seconds.

Events¶

Alfred records Kubernetes events with the source alfred. To list them in every namespace:

kubectl get events -A --field-selector source=alfred -o custom-columns=NAMESPACE:.metadata.namespace,TYPE:.type,REASON:.reason,KIND:.involvedObject.kind,NAME:.involvedObject.name

NAMESPACE   TYPE      REASON                                KIND               NAME
ome         Warning   OMENativeUnavailable                  ConfigMap          alfred-config
prod        Normal    FragmentationRecommendationProduced   InferenceService   chat

Events are lossy. Repeats on one object merge into one with a count. A burst on one object can lose events, or combine them into one whose message starts with (combined from similar events):. The recommendations ConfigMap holds the full result of each pass.

On an InferenceService¶

Alfred records these events on the InferenceService. kubectl describe inferenceservice doesn't show them, so select them by name:

kubectl get events -n prod --field-selector involvedObject.name=chat,source=alfred -o custom-columns=TYPE:.type,REASON:.reason,MESSAGE:.message

TYPE     REASON                                MESSAGE
Normal   FragmentationRecommendationProduced   defragmentation advisory for prod/chat/engine: OMENativeUnavailable (from node-a, footprint 0 GPUs)

The messages have these forms:

• Advice: <policy> advisory for <namespace>/<name>/<component>: <advisoryReason> (from <fromNode>, footprint <GPUs> GPUs). The footprint is 0, except for NoSurgeHeadroom, where it's the Instance's GPUs.
• Rejection: <policy> recommendation for <namespace>/<name>/<component> rejected: <rejectReason>.
• Withheld rows and requests: <policy> migration request uuid=<requestUUID> for <namespace>/<name>/<component> instance <instance>: <dispatchStatus> (<dispatchReason>). The UUID is empty until Alfred creates the request.

Alfred records these events again in every pass. Advice for several Instances of one component on one node shares one event, because the message has no Instance index.

On Alfred's configuration ConfigMap¶

Alfred records both on its configuration ConfigMap, alfred-config by default (the chart value configMapName), in its namespace. Each replica records its own PolicyReloadFailed, and its message gives the error. kubectl describe configmap shows PolicyReloadFailed, but not OMENativeUnavailable. See Hot reload.

On a node¶

Alfred never cordons, drains or evicts: these events are signals for your repair and maintenance tooling. The leader records them on the node, in the default namespace, and kubectl describe node shows them.

OME GPU occupants are OME pods that request GPUs, and the OME workloads in the node's record. A drained node can still run pods that OME doesn't manage.

Each event fires once per episode, which Alfred tracks in the node's record. The repair events can fire again after the node returns to Clear, and the maintenance events after maintenance is no longer requested. NodeDrainedForRepair can also fire again after the node has been Unknown, and both drained events after an OME GPU occupant returns. A node that rejoins with a new UID starts over.

A new leader records the maintenance events again for nodes still under maintenance. It keeps a node's repair phase from its record when the node's state and conditions haven't changed. With recommendationsConfigMapEnabled: false there's no record, so it records the repair events again too. See Signals and events.

The recommendations ConfigMap¶

After each decision pass, the leader writes its result to the ConfigMap alfred-recommendations in Alfred's namespace. recommendationsConfigMapName changes the name, and recommendationsConfigMapEnabled: false turns the record off. The chart creates the ConfigMap. If it's missing, the leader writes no record until it's back: see Alfred logs that the recommendations ConfigMap is missing.

To read the latest pass:

kubectl get configmap alfred-recommendations -n ome -o jsonpath='{.data.last-cycle\.json}' | jq .

{
  "timestamp": "2026-09-28T10:15:02.418093721Z",
  "mode": "recommend-only",
  "recommendations": [
    {
      "workload": "prod/chat",
      "component": "engine",
      "instance": 3,
      "policy": "defragmentation",
      "reason": "Fragmentation",
      "outcome": "advisory",
      "advisoryReason": "OMENativeUnavailable",
      "fromNode": "node-a",
      "score": 0,
      "scheduling": {
        "schedulerName": "default-scheduler",
        "status": "Unavailable",
        "reason": "ProfileNotConfigured"
      }
    }
  ]
}

This row is from the default install. Instance 3 of the engine component of chat runs on node-a in a fragmented pool. With no executor, Alfred can only advise. The kubectl ome admin recommendations command summarizes the same pass. A row leaves out fields that have no value, except score:

A planned move's score is its benefit less a move cost that aggressiveness weights. See How Alfred picks a move. Defragmentation advice scores 0 or less. A node-health row, advice included, scores the InferenceService's alfred.ome.io/priority annotation, 0.5 by default.

Outcomes¶

In recommend-only mode, every row is advisory, except the rows of requests left in the journal from execute mode. A request's row appears in every pass until the request ends, and once more in the pass that sees it end.

When maintenance windows are configured, the defragmentation policy proposes planned moves inside them, and emergencies at any time. See Maintenance windows.

Advisory reasons¶

A component that's OMENative only through its runtime counts as a RawDeployment. For the fix, see An OMENative component shows RawDeploymentMigrationUnsupported.

Scheduling results¶

Alfred adds scheduling to each OMENative row for one Instance. It records which scheduler would place the Instance and, when you set up simulation, where the Instance would go. In execute mode with migration.apiVersion: v1, rows have no scheduling: the dispatcher simulates each move just before it requests it.

Simulation needs the PodGroup CRD, which no OME chart installs. See Gang scheduling. Simulation works without migration.apiVersion. With only simulation.configMapName set, Alfred simulates advice rows, OMENativeUnavailable included, and records their placements. The rows still get no hintTargets. To set up profiles, see Scheduling profiles.

Arbiter reasons¶

In execute mode with migration.apiVersion: v1, the arbiter checks each executable row, health evacuations first. A rejected row shows the first check it failed, in this order:

The targets are the simulated placements, or else the row's hintTargets. Both caps count every migration in the cluster, including kubectl ome migration requests. See The arbiter.

Dispatch reasons¶

With migration.apiVersion: v1, the dispatcher takes the rows that the arbiter admits. It writes one migration request at a time, and none while another request is waiting or in progress in the cluster, including one from kubectl ome migration. A maxInFlightMigrations above 1 still allows only one migration at a time: it lets the dispatcher try the next admitted move when one fails its re-check.

An admitted move that Alfred doesn't request is withheld, with a dispatchReason. These checks cover the whole pass, and withhold every admitted move:

Just before it writes a request, Alfred re-checks the move against a fresh observation. A move that fails is withheld with one of these reasons, and the dispatcher tries the next:

Alfred records the move in the journal before it writes the request. These reasons describe the write:

After Alfred tries a new move, every other admitted row still withheld shows SerialDispatchLimit, even a row that failed a re-check earlier in the pass. So does the tried row while its journal entry is prepared, though the entry keeps its own reason. When no move passes its re-check, each row keeps the reason it failed.

A prepared request left from an earlier pass is retried before any new move, but only within migration.acknowledgementTimeout of its createdAt. With the defaults, the next pass comes later than that, so the request usually stalls with AcknowledgementTimeout instead. If a retried request fails its re-check, every admitted move is withheld with that reason.

Node records¶

A node.<node> key holds a JSON record with these fields:

Alfred deletes a node's record when the node is Clear with no maintenance requested, or leaves the cluster. See Health states.

The dispatch-state ConfigMap¶

With migration.apiVersion: v1, the chart creates the ConfigMap alfred-dispatch-state in Alfred's namespace. The name is fixed. It's Alfred's journal: Alfred records each request there before it writes the request to the InferenceService. In every decision pass, in either mode, the leader updates each entry from what it observes. In recommend-only mode, it writes no requests.

Its one key, state.json, holds version (v1), entries and, after a failure, backoffUntil, when the failure backoff ends. A document Alfred can't read, for example one with an unknown field, gives JournalUnavailable, and the log doesn't say why.

To read the journal:

kubectl get configmap alfred-dispatch-state -n ome -o jsonpath='{.data.state\.json}'

{"version":"v1","entries":[]}

This journal is empty. Each entry has these fields:

Journal states¶

The manager's InferenceReplica controller acknowledges Alfred's requests. With --enable-inferencereplica-controller=false, the first request stalls with AcknowledgementTimeout and blocks every later one.

A failed or stalled entry sets backoffUntil to migration.failureBackoff (default 5 minutes) later, and Alfred withholds moves with FailureBackoff until then. Only completed and failed end an entry. Until then, a stalled entry withholds every new move with UnresolvedRequest, and some stalled entries never end: see Handling the journal.

Handling the journal¶

With Kustomize, config/alfred/dispatch-state.yaml creates the journal. Create it once, on a cluster with no dispatch history:

kubectl create -f config/alfred/dispatch-state.yaml

configmap/alfred-dispatch-state created

Never kubectl apply or kubectl replace the file, and keep it out of Kustomize and GitOps syncs. If kubectl create fails with AlreadyExists, stop and inspect the existing journal.

Some stalled entries never end: a request whose write never reached the InferenceService, or whose InferenceService or InferenceReplica was deleted or replaced. Alfred never cancels a request, so such an entry blocks new moves until you remove it from entries in state.json. A request is stalled shows how to tell such an entry from a request that can still end, and how to remove it. When you turn migration back on, Alfred adopts the journal it finds.

Related pages¶

• Alfred: what Alfred does, its two modes and how its replicas share the work.
• Alfred policies: how the policies pick moves, and how the arbiter and the dispatcher check them.
• Run Alfred in recommend-only mode: install Alfred and read its output.
• Let Alfred migrate Instances: turn on planned moves and execute mode.
• Alfred configuration: every config.yaml key, flag and chart value, with its default.
• kubectl ome admin: read Alfred's latest recommendations from the command line.
• kubectl ome migration: show an InferenceService's migrations and request one.
• Migration and transient scale: how the manager carries out a migration request.