Run Alfred in recommend-only mode Unreleased: coming in v1.3

Install Alfred and read the Instance moves it recommends when free GPUs fragment or a node turns unhealthy, then tune its policies.

Alfred, OME's GPU cluster caretaker, recommends migrations that reclaim fragmented GPUs or move Instances off unhealthy nodes. It starts in recommend-only mode: it reports these moves and changes no workload, so you can review its advice before you let it act. Alfred is alpha. You turn it on by installing its own chart, ome-alfred, next to OME. Then you read its recommendations, events and metrics, and tune a policy.

What Alfred recommends¶

Alfred runs two policies:

The defragmentation policy looks for GPU pools, groups of nodes with the same kind of GPU, whose free GPUs are scattered in pieces too small for larger workloads.
The node-health policy looks for Instances on unhealthy nodes, by default nodes whose GpuUnhealthy condition is True. OME doesn't set this condition, so your GPU health checks must. Once you configure maintenance triggers, the policy also looks for nodes marked for planned maintenance.

Alfred policies explains how they choose.

Step 1: Install Alfred¶

Save the image settings in a values file:

values.yaml

global:
  hub: registry.example.com/ome
image:
  tag: v1.3.0

The chart then runs registry.example.com/ome/alfred:v1.3.0 instead of its unpublished default, ghcr.io/moirai-internal/alfred:latest. If your registry needs credentials, list your pull secrets under global.imagePullSecrets, as name entries.

Install the chart into OME's namespace, ome, where kubectl ome looks for Alfred by default:

Shell

helm install ome-alfred oci://ghcr.io/moirai-internal/charts/ome-alfred \
  --version 1.3.0 --namespace ome -f values.yaml

Helm installs the release and prints its status. The chart creates the Deployment ome-alfred, with three replicas, and two ConfigMaps: alfred-config, which holds Alfred's configuration, and alfred-recommendations, where Alfred writes its recommendations. The chart's notes suggest looking for RecommendationWithheld events, but Alfred records those only in execute mode.

Install one Alfred per cluster. Its ClusterRole and ClusterRoleBinding have fixed names, so a second release in another namespace fails to install.

Wait for Alfred's pods:

Shell

kubectl rollout status deployment/ome-alfred -n ome

Output

deployment "ome-alfred" successfully rolled out

If the rollout stalls, see The pods don't start.

One replica, the leader, runs the policies and writes the recommendations. If it goes away, another replica takes over. See which pod leads:

Shell

kubectl get lease alfred.ome.io -n ome -o jsonpath='{.spec.holderIdentity}'

Output

ome-alfred-6d8f7c9b54-2xkqp_0b6c5f2e-4c1d-4f7e-9a53-2d8e6b1f0c47

The part before the underscore is the leader's pod name.

Step 2: Read its recommendations¶

The leader writes its first recommendations within one decision interval, 5 minutes by default. Read them with kubectl ome:

Shell

kubectl ome admin recommendations

Output

SUBJECT                    EVIDENCE            DETAIL
Alfred recommendations     Reported            Recent
Source namespace           ome                 Config and cycle sources
Config source              alfred-config       config.yaml
Record source              last-cycle.json     alfred-recommendations
ConfigMap config           Available           recommend-only
Latest cycle               Available           recommend-only
Rows                       2 shown/2 scanned   0 invalid; 0 omitted
team-a/llama-8b/engine#0   advisory            defragmentation/Fragmentation
team-b/qwen-32b/engine#0   advisory            defragmentation/Fragmentation
Advisory evidence          Not authorization   Convergence unverified
Executability              Unverifiable        Not persisted in cycle
Scope                      Latest cycle only   Node/scheduling data omitted

The first rows show where the command read the recommendations, the mode, and whether they're Recent: written within the last two decision intervals. Then come the recommendations: the InferenceService, component and Instance, the outcome, and the policy with its reason. Add -o wide to see their advisory reasons. kubectl ome admin explains the output.

In recommend-only mode, the outcome is always advisory. With the chart's defaults, the rows also name no target node, and their advisory reason explains what blocks the move. Alfred moves only OMENative Instances, so the reason depends on the component's deployment mode:

Deployment mode	Advisory reason
RawDeployment	`RawDeploymentMigrationUnsupported`, with one row per flagged pod, numbered from 0 in the order of the pods' names
`MultiNode` (deprecated)	`LWSMigrationUnsupported`
OMENative	`OMENativeUnavailable`, until you set the chart's `migration.apiVersion`, as Let Alfred migrate Instances does
OMENative, but not steady, such as during a rollout	`OMENativeStateIneligible` or `OMENativeObservationInvalid`

A component that's OMENative only through its runtime counts as a RawDeployment: see An OMENative component shows RawDeploymentMigrationUnsupported. Advisory reasons lists the others. The rows show what the policies propose. In execute mode, the arbiter's caps and cooldowns also hold moves back, as Safety bounds explains.

To see all the fields, read the recommendations ConfigMap:

Shell

kubectl get configmap alfred-recommendations -n ome -o jsonpath='{.data.last-cycle\.json}' | jq .

Output

{
  "timestamp": "2026-09-28T10:15:02.418377251Z",
  "mode": "recommend-only",
  "recommendations": [
    {
      "workload": "team-a/llama-8b",
      "component": "engine",
      "instance": 0,
      "policy": "defragmentation",
      "reason": "Fragmentation",
      "outcome": "advisory",
      "advisoryReason": "RawDeploymentMigrationUnsupported",
      "fromNode": "gpu-node-3",
      "score": 0
    },
    {
      "workload": "team-b/qwen-32b",
      "component": "engine",
      "instance": 0,
      "policy": "defragmentation",
      "reason": "Fragmentation",
      "outcome": "advisory",
      "advisoryReason": "OMENativeUnavailable",
      "fromNode": "gpu-node-5",
      "score": 0,
      "scheduling": {
        "schedulerName": "default-scheduler",
        "status": "Unavailable",
        "reason": "ProfileNotConfigured"
      }
    }
  ]
}

On OMENative rows, scheduling is Unavailable with the reason ProfileNotConfigured until you set up simulation, as Set up scheduler simulation for Alfred shows. The recommendations ConfigMap describes the fields.

Alfred also records the rows as events:

Shell

kubectl get events -A --field-selector source=alfred -o custom-columns=NAMESPACE:.metadata.namespace,TYPE:.type,REASON:.reason,KIND:.involvedObject.kind,NAME:.involvedObject.name

Output

NAMESPACE   TYPE      REASON                                KIND               NAME
ome         Warning   OMENativeUnavailable                  ConfigMap          alfred-config
team-a      Normal    RawDeploymentMigrationUnsupported     InferenceService   llama-8b
team-b      Normal    FragmentationRecommendationProduced   InferenceService   qwen-32b

The Warning OMENativeUnavailable is expected until you set migration.apiVersion. kubectl describe doesn't list these events, so use kubectl get events. Alfred merges and rate-limits repeated events, so the ConfigMap is the complete record. Events lists the reasons.

Step 3: Check its metrics¶

All replicas serve Prometheus metrics on port 8080, at /metrics. Only the leader publishes the decision metrics, such as the recommendation counts, so forward a local port to the leader's pod:

Shell

LEADER=$(kubectl get lease alfred.ome.io -n ome -o jsonpath='{.spec.holderIdentity}' | cut -d_ -f1)
kubectl port-forward -n ome "pod/$LEADER" 8080:8080

Output

Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Leave it running, and read a few of Alfred's metrics from another terminal:

Shell

curl -s localhost:8080/metrics | grep -E '^alfred_(cluster_fragmentation_score|fragmentation_observed|leader_status|omenative_unavailable|policy_reload_total|recommendations_produced_total)'

Output

alfred_cluster_fragmentation_score 0
alfred_fragmentation_observed{pool="NVIDIA-H100-80GB-HBM3",size="1"} 0
alfred_fragmentation_observed{pool="NVIDIA-H100-80GB-HBM3",size="2"} 0.25
alfred_fragmentation_observed{pool="NVIDIA-H100-80GB-HBM3",size="4"} 1
alfred_fragmentation_observed{pool="NVIDIA-H100-80GB-HBM3",size="8"} 1
alfred_leader_status{pod="ome-alfred-6d8f7c9b54-2xkqp"} 1
alfred_omenative_unavailable 1
alfred_policy_reload_total{outcome="success"} 1
alfred_recommendations_produced_total{component="engine",executable="false",policy="defragmentation",reason="Fragmentation",workload="team-a/llama-8b"} 4
alfred_recommendations_produced_total{component="engine",executable="false",policy="defragmentation",reason="Fragmentation",workload="team-b/qwen-32b"} 4

Metric	What it shows
`alfred_cluster_fragmentation_score`	The highest pool score, from 0 to 1. It counts only the fragmentation that moving Instances could fix, so it stays 0 until you set `migration.apiVersion`.
`alfred_fragmentation_observed`	By GPU pool and Instance size, the share of the pool's free GPUs that Instances of that size can't use.
`alfred_leader_status`	1 on the leader, 0 on the other replicas.
`alfred_omenative_unavailable`	1 while OMENative migration is unavailable, as with the chart's defaults.
`alfred_policy_reload_total`	The configuration loads, by `outcome`: `success` or `failure`.
`alfred_recommendations_produced_total`	The rows that the policies produced, pass after pass. `executable="false"` marks advice.

The defragmentation policy still reports rows while the score is 0. With the chart's defaults, it flags a pool whose demand-weighted fragmentation is above its fragmentationThreshold, 0.25 by default. How Alfred scores a pool explains both figures.

To scrape Alfred with the Prometheus Operator, set metrics.serviceMonitor.enabled: true in values.yaml. The chart then creates a ServiceMonitor that scrapes all replicas through the Service ome-alfred-metrics. The pods also carry prometheus.io/scrape annotations. Alfred metrics and events lists all of Alfred's metrics.

Step 4: Tune a policy¶

You configure Alfred through the chart's alfredConfig value, which the chart writes to the ConfigMap alfred-config. Alfred configuration lists the settings.

For example, to have the defragmentation policy flag only more fragmented pools, raise its fragmentationThreshold from 0.25 to 0.4 in values.yaml:

values.yaml

global:
  hub: registry.example.com/ome
image:
  tag: v1.3.0
alfredConfig:
  policies:
    defragmentation:
      fragmentationThreshold: 0.4

Keep the image settings in the file: with -f, helm upgrade starts again from the chart's defaults. Under alfredConfig, set only what you change. Helm merges maps into the chart's defaults key by key, but a list, such as earlyTickOn, replaces the chart's list whole.

Upgrade the release:

Shell

helm upgrade ome-alfred oci://ghcr.io/moirai-internal/charts/ome-alfred \
  --version 1.3.0 --namespace ome -f values.yaml

Helm updates alfred-config and replaces Alfred's pods, which start with the new configuration. Check that the replicas loaded it:

Shell

kubectl logs -n ome -l control-plane=ome-alfred --tail=-1 --prefix | grep 'config reload'

kubectl prints the matching lines, prefixed with the pod's name. A replica that applied the configuration logs config reloaded, with the mode it runs in. If one logs config reload failed; keeping last-known-good, see A configuration change doesn't take effect. After the next decision pass, kubectl ome admin recommendations shows fewer defragmentation rows, or none.

To try a change without an upgrade, edit the ConfigMap with kubectl edit configmap alfred-config -n ome. Alfred applies the edit within seconds, without a restart. With Helm 3, the next helm upgrade writes the chart's configuration back over it, so keep your settings in values.yaml.

Troubleshooting¶

The pods don't start¶

List Alfred's pods:

Shell

kubectl get pods -n ome -l control-plane=ome-alfred

kubectl lists the pods with their status. ErrImagePull or ImagePullBackOff means that the nodes can't pull the image. Check global.hub and image.tag against the image you pushed, and add a pull secret to global.imagePullSecrets if your registry needs one. Pending means that no node has room for a pod, which requests 100m of CPU and 256Mi of memory: kubectl describe pod shows why.

The command reports Unavailable or Stale¶

The first rows of kubectl ome admin recommendations show the cause:

What you see	Cause and fix
`KeyAbsent` on the `Latest cycle` row	The leader hasn't finished its first pass. Wait one decision interval, 5 minutes by default.
`Stale` on the first row	No leader has finished a pass for two decision intervals. Check the Lease and the logs, as below.
`NotFound` on the `ConfigMap config` row	The command looked for Alfred in the wrong namespace. Add `--alfred-namespace` with the namespace of Alfred's release.
`NotFound` on the `Latest cycle` row	`alfred-recommendations` is missing. See Alfred logs that the recommendations ConfigMap is missing.

If KeyAbsent or Stale persists, check that the Lease has a holder, as Step 1 shows, and read the replicas' logs:

Shell

kubectl logs -n ome -l control-plane=ome-alfred --tail=-1 --prefix

kubectl prints the replicas' logs. A repeated observation pass failed error means that the replica can't read the cluster, and the rest of the line says why. A single initial observation pass failed error while a pod starts is harmless on its own.

Alfred recommends nothing¶

The command reports Empty when the leader's pass found nothing to report, which is normal on a healthy cluster. The defragmentation policy flags a pool when its demand-weighted fragmentation is above fragmentationThreshold, and alfred_fragmentation_observed shows how fragmented the pools are. The node-health policy flags Instances on nodes whose GpuUnhealthy condition is True, or on nodes that match a maintenance trigger you configure. Alfred configuration shows how to change the threshold, conditions and triggers.

An OMENative component shows RawDeploymentMigrationUnsupported¶

Alfred reads a component's deployment mode from the InferenceService alone. A component that's OMENative only through its runtime, by the runtime's leader or worker or its ome.io/deploymentMode annotation, is a RawDeployment to Alfred, which never moves it. Set spec.deploymentMode: OMENative in the InferenceService, as For the whole InferenceService shows. The field also makes a router OMENative, unless the router sets its own mode.

If the runtime sets the mode with the ome.io/deploymentMode annotation, set that annotation on the component in the InferenceService instead, as For one component shows.

A configuration change doesn't take effect¶

Look for PolicyReloadFailed events. Alfred records one when it rejects a configuration, such as a fragmentationThreshold of 1.5:

Shell

kubectl get events -n ome --field-selector reason=PolicyReloadFailed

Output

LAST SEEN   TYPE      REASON               OBJECT                    MESSAGE
2m10s       Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5
2m10s       Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5
2m9s        Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5
2m4s        Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5
104s        Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5
83s         Warning   PolicyReloadFailed   configmap/alfred-config   alfred-config rejected, keeping last-known-good: policies.defragmentation.fragmentationThreshold must be in [0, 1], got 1.5

The running replicas and the new pods that the upgrade starts all record the event. The running replicas keep their last good configuration, but a new pod has none, so it runs Alfred's built-in defaults: recommend-only mode, with both policies on. Once the upgrade finishes, all replicas run these defaults until you fix the value and upgrade again.

If there are no such events, check that your setting reached alfred-config. Helm accepts values that the chart ignores, such as policies at the top level instead of under alfredConfig. Print what Alfred reads:

Shell

kubectl get configmap alfred-config -n ome -o jsonpath='{.data.config\.yaml}'

Check that your setting is in the config.yaml document it prints.

The command reports Partial¶

The command shows at most 200 rows, and the Rows row counts the ones it left out. It also leaves out rows about a whole component, which have instance -1 in the ConfigMap, and counts them as invalid, with an InvalidRows issue. This is a known bug: read the ConfigMap, as Step 2 shows, to see them all.

Alfred logs that the recommendations ConfigMap is missing¶

The leader logs recommendations ConfigMap missing; record disabled (the chart pre-creates it) when alfred-recommendations is gone. Alfred can't recreate it, so run your helm upgrade again. Alfred writes its recommendations again at its next pass.

Clean up¶

Uninstall the chart:

Shell

helm uninstall ome-alfred --namespace ome

Output

release "ome-alfred" uninstalled

Helm deletes everything that the chart created, including Alfred's recommendations. The leader-election Lease alfred.ome.io stays, because Alfred created it. Delete it:

Shell

kubectl delete lease alfred.ome.io -n ome

Output

lease.coordination.k8s.io "alfred.ome.io" deleted

Alfred's events expire after an hour by default. Recommend-only mode left your workloads as they were.

Next steps¶

Set up scheduler simulation for Alfred: predict where the replacement pods of each move would land.
Let Alfred migrate Instances: turn on migration, and let Alfred carry out its moves.
Alfred: how Alfred observes, decides and migrates.
Alfred policies: how the defragmentation and node-health policies choose what to move.
Alfred configuration: the settings in alfredConfig.
Alfred metrics and events: the metrics and events that Alfred records.
kubectl ome admin: the flags and output of kubectl ome admin recommendations.