Collect metrics Unreleased: coming in v1.3

By default, the ome-resources chart runs a Prometheus server, ome-prometheus, that scrapes your InferenceService pods and OME's own components. Two features read from it: canary analysis on OMENative components, when a canary names no other source, and KEDA triggers that point at http://ome-prometheus.ome.svc:9090. It keeps 30 minutes of data on a temporary volume: enough for those decisions, too little for dashboards or long-term monitoring. For those, use your own Prometheus, next to the bundled one or in its place.

What it scrapes¶

Prometheus runs three scrape jobs, every 15 seconds by default (prometheus.scrapeInterval):

The chart lets Prometheus read pods, Services and Endpoints in all namespaces. To scrape more, add jobs to prometheus.extraScrapeConfigs. Limit them to the namespaces that they need: a cluster-wide job makes Prometheus hold all the cluster's pods in memory.

InferenceService pods¶

The job finds pods by the label ome.io/inferenceservice, which OME sets on all InferenceService pods. It scrapes a pod while the pod is running and ready.

By default, it scrapes all these pods except the ones annotated prometheus.io/scrape: "false". On large or shared clusters, set prometheus.requireScrapeAnnotation: true to scrape only the pods annotated prometheus.io/scrape: "true". Opt pods in with that annotation, not with ome.io/enable-prometheus-scraping: see A target is down.

It scrapes the port in the pod's prometheus.io/port annotation, at the path in prometheus.io/path, or /metrics by default. Most catalog runtimes set these annotations in their engineConfig.annotations, and yours can too.

It adds these labels to the series that it scrapes:

KEDA triggers and canary queries select series by these labels. A canary query can match revision_hash to compare the new revision's pods with the stable revision's.

OME's components¶

The ome-control-plane job reads the same port and path annotations, and adds the namespace and pod labels:

Servers and metrics describes the controller's metrics endpoint, and Controller metrics the canary metrics on it.

Step 1: Check the targets¶

Forward a local port to the Prometheus Service:

kubectl port-forward -n ome service/ome-prometheus 9090:9090

Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

From another terminal, list the active targets:

curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | [.scrapePool, .labels.pod // "-", .health] | @tsv'

Each line shows a target's job, pod and health: up, down, or unknown before its first scrape. Expect lines for the pods that What it scrapes covers, and one ome-prometheus line with - for the pod. If a pod is missing or a target is down, see Troubleshooting.

List the labels of the InferenceService targets:

curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.labels.inferenceservice) | .labels | [.namespace, .inferenceservice, .component, .revision_hash // "-", .pod] | @tsv'

Each line shows a pod's namespace, InferenceService, component, revision hash (- if none) and name.

Step 2: Tune retention, storage and scope¶

By default, Prometheus keeps 30 minutes of data, up to 6GB, in an 8Gi emptyDir volume that's lost when the pod is replaced. Its 3Gi memory limit suits about 500 InferenceService pods. On a larger fleet, scrape less before you add memory.

This example keeps two hours of data on a volume that survives restarts, and scrapes the InferenceService pods of one namespace:

prometheus:
  retention: 2h
  retentionSize: 12GB
  persistence:
    enabled: true
    size: 16Gi
  scrapeNamespaces:
    - llama-demo

• persistence replaces the emptyDir with a 16Gi PersistentVolumeClaim, ome-prometheus, from the default StorageClass. Key values lists the claim's other settings.
• retentionSize: 12GB fits the larger volume. Keep it below 80 to 85% of the volume's size, as the default 6GB does for 8Gi.
• scrapeNamespaces limits the InferenceService job to llama-demo. The ome-control-plane job still scrapes the namespace ome.

To cut memory further, set requireScrapeAnnotation: true, or drop series with metricRelabelConfigs. This value keeps two of vLLM's series and drops the rest, so first check which series your triggers and queries read:

prometheus:
  metricRelabelConfigs:
    - source_labels: [__name__]
      regex: "vllm:num_requests_waiting|vllm:num_requests_running"
      action: keep

Add these values to your values file: helm upgrade resets any value that's missing from it. Upgrade with your current chart version, from the CHART column of helm list -n ome:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Helm reports that the release ome has been upgraded. Wait for the new Prometheus pod:

kubectl rollout status deployment ome-prometheus -n ome

When the new pod is running, the command ends with:

deployment "ome-prometheus" successfully rolled out

Print the arguments of the prometheus container, one per line:

kubectl get deployment ome-prometheus -n ome \
  -o jsonpath='{range .spec.template.spec.containers[?(@.name=="prometheus")].args[*]}{@}{"\n"}{end}'

With the values above and other values at their defaults, the output is:

--config.file=/etc/prometheus/prometheus.yml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention.time=2h
--storage.tsdb.retention.size=12GB
--web.enable-lifecycle
--web.listen-address=:9090

Restart the port-forward, which stopped with the old pod, and rerun the labels command from Step 1. Its first column now reads llama-demo.

Use your own Prometheus¶

Run your own Prometheus next to the bundled one for dashboards and long-term storage, or in its place. ServiceMonitors, PodMonitors and the chart's alerts need a Prometheus that the Prometheus Operator runs. To replace the bundled one, set up scraping first, then turn it off.

Scrape InferenceService pods¶

Your Prometheus needs to add the labels that queries select by. Any Prometheus can copy the bundled scrape job, and on a Prometheus Operator stack, OMENative components can use OME's PodMonitors.

helm template ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 --show-only templates/prometheus/configmap.yaml

ome:
  podMonitor:
    labels:
      release: prometheus
    relabelings:
      - sourceLabels: ["__meta_kubernetes_pod_label_ome_io_inferenceservice"]
        targetLabel: inferenceservice
      - sourceLabels: ["__meta_kubernetes_pod_label_component"]
        targetLabel: component
      - sourceLabels: ["__meta_kubernetes_pod_label_ome_io_revision_hash"]
        targetLabel: revision_hash

To check the result, run the labels command from Step 1 against your Prometheus.

Scrape OME's components¶

On a Prometheus Operator stack, turn on the monitors in the OME's components table, and add the labels that your Prometheus selects to their additionalLabels. The controller's ServiceMonitor, ome-controller-manager, needs the Prometheus Operator's monitoring.coreos.com/v1 CRDs. Key values lists its settings.

A Prometheus that scrapes pods by their prometheus.io/scrape annotation finds the controller, model agent, quota manager and Alfred without monitors. Use one of the two ways, or it scrapes these pods twice.

Turn off the bundled Prometheus¶

Point canary analysis and KEDA at your Prometheus in the same upgrade that sets prometheus.enabled: false. This example uses a Prometheus behind the Service prometheus in the namespace monitoring:

prometheus:
  enabled: false
ome:
  controller:
    canaryAnalysis:
      bundledPrometheusAddress: "http://prometheus.monitoring.svc:9090"
  metricProviders:
    cluster-prometheus:
      serverAddress: "http://prometheus.monitoring.svc:9090"

• bundledPrometheusAddress is the source for canaries that name none, and can be any Prometheus. If it's empty, the chart still renders the bundled address, so those canaries' samples are all inconclusive. To send a bearer token or headers, name a binding in ome.controller.canaryAnalysis.defaultProvider instead.
• ome.metricProviders binds the names that canaries and alpha AutoscalerPolicy triggers give in providerRef: see Metric provider bindings. Point the bindings that named the bundled Prometheus at yours.
• In your InferenceServices, change the serverAddress of the KEDA triggers that point at ome-prometheus.

The upgrade deletes the bundled Prometheus and its data, including the claim ome-prometheus if the chart created it. A claim that you named in persistence.existingClaim stays, and so do your ServiceMonitors, PodMonitors and alert rules.

Upgrade the release with the helm upgrade command from Step 2. Then print the controller's canary settings:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.canaryAnalysis}'

{
  "bundledPrometheusAddress": "http://prometheus.monitoring.svc:9090",
  "defaultProvider": "",
  "queryTimeout": "5s",
  "maxConcurrency": 8,
  "cacheTTL": "2m"
}

The upgrade also restarts the controller manager, which then uses the new addresses.

Key values¶

The prometheus values configure the bundled Prometheus:

The serviceMonitor values configure the controller's ServiceMonitor:

Troubleshooting¶

A pod is missing from the targets¶

Check that the pod is running and ready. Then check its prometheus.io/scrape annotation, and the requireScrapeAnnotation and scrapeNamespaces values: see What it scrapes.

A target is down¶

List the targets whose last scrape failed, with the URL that Prometheus scraped and the error:

curl -s localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | select(.health == "down") | [.scrapePool, .scrapeUrl, .lastError] | @tsv'

Compare a target's port and path with its pod's annotations, and with the port that the pod serves metrics on.

A port of 9091 or 9088, where no OME container serves metrics, comes from the annotation ome.io/enable-prometheus-scraping: "true", or from the chart value ome.metricsaggregator.enablePrometheusScraping: "true". Remove the annotation, or set the value back to "false", its default. Then opt the pod in with prometheus.io/scrape: "true".

Prometheus keeps restarting¶

A Prometheus that runs out of memory can run out again as it restarts and reloads recent data. Scrape less, as in Step 2, then raise its memory in prometheus.resources.

InferenceServices stop updating¶

Look for a binding without serverAddress in the logs of all controller replicas:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'has no serverAddress'

Each line names the binding, in a fails to load metricProviders config or fails to load autoscalerPolicy config error. Add the binding's serverAddress and upgrade again.

Next steps¶

• Set up alerting: turn on the chart's alerts, on a Prometheus that the Prometheus Operator runs.
• Canary metric analysis: write the queries that gate a canary's steps.
• Component autoscaling: scale a component on a prometheus trigger.