Set up alerting Unreleased: coming in v1.3

The ome-resources chart ships five Prometheus alerts for OME failures that should page someone at once. They cover a controller that's down or stalled, a failing admission webhook, an InferenceService with no healthy backend, and rollouts that fail across many InferenceServices. They're off by default: turn them on once your Alertmanager routes their severity label to someone.

As shipped, OMEWebhookFailing never fires, and OMEServiceUnavailable fires only if your router exports the series that it reads.

Step 1: Turn on the alerts¶

A Prometheus uses the chart's PrometheusRule and ServiceMonitor only when its selectors match their labels and the namespace ome. Print the selectors of each Prometheus:

kubectl get prometheus -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,\
RULES:.spec.ruleSelector,RULE-NAMESPACES:.spec.ruleNamespaceSelector,\
MONITORS:.spec.serviceMonitorSelector,MONITOR-NAMESPACES:.spec.serviceMonitorNamespaceSelector"

Read the output this way:

By default, a kube-prometheus-stack Prometheus selects objects by the label release: map[matchLabels:map[release:kube-prometheus-stack]]. Add the labels that your Prometheus selects to both objects with additionalLabels, and turn both on:

prometheusRule:
  enabled: true
  additionalLabels:
    release: kube-prometheus-stack
  runbookUrlBase: "https://runbooks.example.com/ome"
serviceMonitor:
  enabled: true
  additionalLabels:
    release: kube-prometheus-stack

runbookUrlBase is optional: see Runbook links. For the ServiceMonitor's other settings, see Key values in Collect metrics.

helm upgrade resets every value that isn't in the file to the chart's default, so keep your other install values in it. Upgrade the release with the chart version that you already run, 1.3.0 here:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that the release ome has been upgraded. It adds the PrometheusRule ome-p0-alerts and the ServiceMonitor ome-controller-manager, and leaves the controller's pods as they are.

Step 2: Check the rules¶

List the alerts in the PrometheusRule:

kubectl get prometheusrule ome-p0-alerts -n ome \
  -o jsonpath='{range .spec.groups[*].rules[*]}{.alert}{"\n"}{end}'

OMEControllerDown
OMEWebhookFailing
OMEServiceUnavailable
OMEMassRolloutFailure
OMEReconcileStalled

To check that Prometheus loaded the rules, forward a local port to it:

kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

Open http://localhost:9090/alerts in a browser. The page lists the group ome.p0 with the five alerts. If the group is missing, see Prometheus doesn't load the rules.

Then check that Prometheus scrapes the controller: run the query up{pod=~"ome-controller-manager.*"} in the Prometheus UI. It returns one series per controller replica, with the value 1. If it returns nothing, OMEControllerDown fires after 5 minutes: see OMEControllerDown fires as soon as you turn it on.

The five alerts¶

Each alert fires when its condition holds for the time in the table. Route the alerts in Alertmanager on their labels: tier: p0, the component in the table, and severity, which is critical by default.

OMEControllerDown¶

No controller replica is working, so no InferenceService deploys, scales or rolls out. The alert fires when Prometheus can't scrape any replica, or when the replicas run but none holds the leader Lease.

When it fires:

1. Check that the pods run: kubectl get pods -n ome -l control-plane=ome-controller-manager. If they run and you've just turned the alerts on, see OMEControllerDown fires as soon as you turn it on.
2. Read every replica's log: kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1. To find the leader, see Tune leader election.
3. If a pod's RESTARTS count went up, read the run that ended: kubectl logs ome-controller-manager-7c9d5b8f4-kx2mq -n ome --previous, with your pod's name. A leader that can't renew the Lease logs Failed to start manager, with the error leader election lost, and restarts.

absent(up{pod=~"ome-controller-manager.*"}) == 1
or max(up{pod=~"ome-controller-manager.*"}) == 0
or max(leader_election_master_status{pod=~"ome-controller-manager.*"}) == 0

OMEWebhookFailing¶

The alert counts the admission requests that OME's webhooks answer with a 5xx status. The webhooks never answer that way, so as shipped the alert never fires. To leave it out, set prometheusRule.rules.webhookFailing.enabled to false.

While the webhook can't be reached, the API server rejects InferenceService creates and updates. If that's because no controller replica runs, OMEControllerDown fires.

sum(rate(controller_runtime_webhook_requests_total{pod=~"ome-controller-manager.*",code=~"5.." }[5m]))
/
sum(rate(controller_runtime_webhook_requests_total{pod=~"ome-controller-manager.*"}[5m]))
> 0.1

OMEServiceUnavailable¶

An InferenceService has backends registered with its routers, but no router sees a healthy one, so requests to it fail.

The alert reads router_pool_healthy_count and router_pool_total_count, with the labels namespace and inferenceservice. OME doesn't export them, so the alert fires only if your router does.

When it fires, check the engine pods before the routers: a backend that routers drop is usually an engine that stopped passing its health check. See Troubleshoot an InferenceService.

(max by (namespace, inferenceservice) (router_pool_healthy_count{}) == 0)
and
(max by (namespace, inferenceservice) (router_pool_total_count{}) > 0)

OMEMassRolloutFailure¶

More than half of the rollout groups that finished in the last 15 minutes failed, in at least 3 InferenceServices. Failures across many workloads at once point at the control plane, or at something the workloads share.

The alert counts the blue-green and rolling-update rollout groups of OMENative components. Canary groups don't count. It counts InferenceServices by name, so two with the same name in different namespaces count once.

The InferenceService's namespace is in the label exported_namespace, because namespace holds the controller's. When the alert fires, list the groups that failed with this query in the Prometheus UI:

sum by (exported_namespace, isvc, group) (increase(ome_omenative_rollout_group_transition_total{to="Failed"}[15m])) > 0

For example, {exported_namespace="demo", group="0", isvc="qwen2-5-7b"} is group 0 of the InferenceService qwen2-5-7b, in the namespace demo. Check the rollouts of a few of the listed InferenceServices, as in A change doesn't roll out in Troubleshoot an InferenceService.

(
  sum(rate(ome_omenative_rollout_group_transition_total{pod=~"ome-controller-manager.*",to="Failed" }[15m]))
  /
  sum(rate(ome_omenative_rollout_group_transition_total{pod=~"ome-controller-manager.*",to=~"Idle|Staged|Failed" }[15m]))
  > 0.5
)
and
(
  count(
    sum by (namespace, isvc) (
      rate(ome_omenative_rollout_group_transition_total{pod=~"ome-controller-manager.*",to="Failed" }[15m])
    ) > 0
  ) >= 3
)

OMEReconcileStalled¶

The controller is up and looks healthy, but it's stuck: its work queues grow while it finishes almost no reconciles. The alert fires when these conditions hold for 10 minutes:

• More than minQueueDepth items wait in the work queues.
• The controller finished at most maxReconcileRate reconciles per second over the last 10 minutes, counting failed ones.
• The queue depth rose over the last 10 minutes.

The alert adds up the work of all OME's controllers, so one stuck controller fires it only when the others are nearly idle too.

When it fires, read every replica's log for a blocked call to the API server or a reconcile that never finishes: kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1. Then restart the controller: kubectl rollout restart deployment/ome-controller-manager -n ome. If it's slow rather than stuck, see Tell whether the manager is throttled.

(sum(workqueue_depth{pod=~"ome-controller-manager.*"}) > 1)
and
(sum(rate(controller_runtime_reconcile_total{pod=~"ome-controller-manager.*"}[10m])) <= 0.01)
and
(deriv(sum(workqueue_depth{pod=~"ome-controller-manager.*"})[10m:]) > 0)

Settings¶

All the values are under prometheusRule.

Each alert has its own values under rules.<key>. enabled is true by default, and duration is how long the condition holds before the alert fires.

Keep rules.controllerDown.duration well above a minute. After the leader's pod goes away, no replica leads until the Lease expires, about a minute later with the chart's settings.

Runbook links¶

Each alert's runbook_url annotation is runbookUrlBase, then # and the alert's name in lowercase. With the value from Step 1, OMEControllerDown links to https://runbooks.example.com/ome#omecontrollerdown. Give your runbook a section for each of the five anchors, or set runbookUrlBase to this page's URL: its alert headings have the same ids.

Scope the alerts to one cluster¶

When one Prometheus holds several clusters' series, the default selectors match every cluster's controller, and the alerts combine them. OMEControllerDown, for example, fires only when no cluster has a working controller. Add a label that tells the clusters apart to both selectors. Your value replaces the default, so keep the pod matcher:

prometheusRule:
  selectors:
    controlPlane: 'pod=~"ome-controller-manager.*",cluster="prod-us-1"'
    router: 'cluster="prod-us-1"'

Don't set selectors.controlPlane to an empty string: the chart would then render invalid PromQL.

Troubleshooting¶

Helm can't find the PrometheusRule kind¶

helm upgrade fails with an error that includes no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", or the same for ServiceMonitor. The cluster doesn't have the Prometheus Operator's CRDs. Install the Prometheus Operator before you turn on the alerts, or leave prometheusRule.enabled and serviceMonitor.enabled at false.

Prometheus doesn't load the rules¶

The PrometheusRule ome-p0-alerts exists, but the Prometheus's alerts page doesn't list the group ome.p0. Print the rule's labels, and compare them with the selectors from Step 1:

kubectl get prometheusrule ome-p0-alerts -n ome --show-labels

If they don't match, change additionalLabels and upgrade again, add the labels that the namespace selector wants to the namespace ome, or change the Prometheus's selectors.

OMEControllerDown fires as soon as you turn it on¶

Five minutes after you turn the alerts on, OMEControllerDown fires while the controller runs. Prometheus doesn't scrape the controller, or its series lack a pod label that matches selectors.controlPlane. Check that the ServiceMonitor exists:

kubectl get servicemonitor ome-controller-manager -n ome --show-labels

If kubectl prints a NotFound error, set serviceMonitor.enabled to true and upgrade again. If it shows the ServiceMonitor, check that the Prometheus's serviceMonitorSelector and serviceMonitorNamespaceSelector match its labels and the namespace ome, as in Step 1. If your Prometheus scrapes the controller another way, set selectors.controlPlane to match the label that holds the pod's name.

Clean up¶

To remove the alerts, set prometheusRule.enabled and serviceMonitor.enabled to false in your values file. Keep serviceMonitor.enabled if something else, such as a dashboard, uses the controller's series. Then upgrade the release as in Step 1:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm deletes the PrometheusRule ome-p0-alerts, and the ServiceMonitor ome-controller-manager if you turned it off. Check that the rule is gone:

kubectl get prometheusrule ome-p0-alerts -n ome

Error from server (NotFound): prometheusrules.monitoring.coreos.com "ome-p0-alerts" not found

Next steps¶

• Collect metrics: tune the Prometheus that the chart deploys, and scrape OME's metrics.
• Troubleshoot an InferenceService: find out why an InferenceService isn't ready or its rollout is stuck.
• Configure the controller: tune leader election and reconcile throughput, which OMEControllerDown and OMEReconcileStalled watch.