Canary metric analysis Unreleased: coming in v1.3

Add analysis to a canary step, and the step moves on when Prometheus says the new revision is healthy. Every interval, OME runs the step's PromQL queries and compares each result with a threshold. The step advances on a sample in which every metric passes, and the canary rolls back when failureLimit samples fail. A sample that can't be judged is inconclusive, and onInconclusive decides what it does. Canary progression covers the steps and their other gates.

Canary works only on components that use the OMENative deployment mode: Deployment modes and OMENative shows how to opt in. A RolloutPolicy, which is alpha, carries the same fields at spec.canary.

Example¶

This canary moves a quarter of the engine's instances to the new revision and checks their error rate before it moves the rest:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: chat
  namespace: prod
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  engine:
    minReplicas: 4
    maxReplicas: 4
  rollout:
    groups:
      - components:
          - engine
        canary:
          steps:
            - capacity: "25%"
              traffic: 10
              pause:
                duration: 10m
              analysis:
                interval: 1m
                initialDelay: 2m
                failureLimit: 3
                metrics:
                  - name: error-rate
                    operator: LT
                    threshold: "0.01"
                    query: |
                      sum(rate(request_errors_total{namespace="{{.Namespace}}", inferenceservice="{{.ISVCName}}", revision_hash="{{.CanaryRevision}}"}[5m]))
                      / sum(rate(requests_total{namespace="{{.Namespace}}", inferenceservice="{{.ISVCName}}", revision_hash="{{.CanaryRevision}}"}[5m]))
            - capacity: "100%"
              traffic: 100

• Step 1 samples about once a minute, from 2 minutes after its new instances are ready.
• It advances on the first passing sample after its 10-minute pause, and rolls the canary back at the third failing sample.
• request_errors_total and requests_total stand for your runtime's error and request counters.
• Step 2 has no gate, so the canary completes once the new revision is ready on all the engine's instances.

Fields¶

Each step sets its own checks, at spec.rollout.groups[].canary.steps[].analysis:

Each metric has these fields, all required:

A pause.duration on the step sets a minimum bake: the step advances on the first passing sample after it. An analysis step ignores a pause without a duration.

The canary sets one source for all its analysis steps, at spec.rollout.groups[].canary.prometheus. Without one, it queries the default source, normally the chart's bundled Prometheus.

This canary queries its own Prometheus, with a token and a tenant header:

spec:
  rollout:
    groups:
      - components:
          - engine
        canary:
          prometheus:
            serverAddress: "http://prometheus.monitoring.svc:9090"
            authRef:
              name: prometheus-bearer
              key: token
            headers:
              X-Scope-OrgID: tenant-a

There are no fields for a CA bundle or a client certificate, so mutual TLS isn't supported.

A running canary keeps the analysis and prometheus blocks it started with, but reads the token Secret and the binding at every sample. An edit to the blocks applies from the next rollout, or when you repin the canary with the alpha kubectl ome rollout repin.

Query template variables¶

A query is a Go template with these variables:

The bundled Prometheus puts these labels, and pod, on the series of InferenceService pods. Another Prometheus needs a scrape configuration that adds them.

Before you gate a step on a query, run it in Prometheus with real label values, and check that it returns a number. Check the targets shows how to reach the bundled Prometheus.

When queries run¶

An analysis starts once the step's new instances are ready. It waits initialDelay, then samples every interval. If the step returns to Pending because it lost instances, the warm-up and the bake start over.

Each query is an instant query at the time of the sample, so [5m] covers the 5 minutes before it. Until the new pods have served requests in that range, a query can return no data, and initialDelay gives them that time. The bundled Prometheus can drop data older than 30 minutes by default (prometheus.retention), so keep ranges within 30 minutes.

How a sample is judged¶

For each metric, OME compares the worst number in the query's result with threshold: the highest for LT and LTE, and the lowest for GT and GTE. So a query that returns one series per pod passes only when every pod passes. A query that returns an empty vector, or a ratio over zero requests, makes the metric inconclusive with the message no data. It's also inconclusive when its query fails or returns a range vector.

At the last step, the canary completes on the first passing sample after scaleDownDelaySeconds, when set.

The alpha kubectl ome rollout promote --override-analysis --yes passes an analysis gate during its warm-up, sampling or bake: see Promote or roll back a canary.

An analysis rollback works like a manual rollback: every instance returns to the stable revision, and the component stays there until a different target revision appears. OME records the rejected revision in rolledBackRevisionHash, and promote and rollback then refuse with action refused: no applicable active rollout or lifecycle work was observed. The analysis starts its rollback only when the stable revision still runs an Instance: when it doesn't, the canary parks Failed instead, as When the stable revision runs no Instance describes.

onInconclusive¶

With Rollback, a Prometheus outage, a missing token Secret or a no data result rolls the canary back. A rollback from Rollback or RollbackOnStall is an analysis rollback, so it too needs a stable revision that runs an Instance.

The analysis stalls when its samples stay inconclusive for longer than the canary's ready timeout, 15 minutes with the chart's defaults.

A Failed canary stays at its step, even after you fix the metrics source, and promote --override-analysis refuses there. A change that gives the component a revision other than the failed canary's and the stable one starts a fresh canary at step 1. You can also roll the canary back with the alpha kubectl ome rollout rollback, then apply your fix, as When a step never becomes ready shows.

When the stable revision runs no Instance¶

A rollback returns every Instance to the stable revision, so a rollback the analysis decides — at failureLimit, on an inconclusive sample with Rollback, or at a stall with RollbackOnStall — needs that revision to still run. While a canary runs, each component with more than one desired Instance keeps one Instance on its stable revision, so the revision normally serves until the canary completes. When any such component in the group runs no live Instance on its stable revision, the fleet has left that revision, and OME doesn't rebuild every Instance onto it on its own:

• The canary parks at Failed at its current step, and status.components.<component>.canary.failed.reason records StableRevisionMissing.
• No revert starts: rolledBackRevisionHash stays empty, and the analysis stops sampling while the canary is parked.
• OME emits the Warning event CanaryStableRevisionEmpty on the InferenceService, once: <primary> is not rolled back on its own: <component> (stable revision <hash>) runs no live Instance to return to; parked Failed at step <N> until a rollback is requested (ome.io/rollout-rollback) or a new target appears. <primary> is the group's router, or else its engine, <component> the component whose stable revision is empty, and the step counts from 0.
• Traffic in status follows the pods that serve: a component whose stable revision runs no pod shows 100% for the canary revision, and a component whose stable revision serves keeps the step's split.

The canary keeps its Instances, and the park holds until you act:

• A rollback is your decision, so it proceeds from the park: the revert rebuilds every Instance of the component onto the stable revision, and the same CanaryStableRevisionEmpty event says so: <primary> rolls back as requested although <component> (stable revision <hash>) runs no live Instance; the revert rebuilds every Instance onto it. Traffic moves to the stable revision once it serves again.
• A change that gives the group a revision other than the parked canary's and the stable one starts a fresh canary at step 1.

A component with a single desired Instance stages in place and keeps no Instance on its stable revision, so an empty stable revision there never parks the canary: its rollback proceeds.

Analysis in status¶

OME records the analysis in status.components.<component>.canary, where <component> is the group's router, or else its engine. The phase is in status.components.<component>.rolloutPhase. While a step waits on its analysis, kubectl ome rollout status shows GATE Analysis, and the promote preview shows the analysis state.

Each entry of metricResults holds the metric's name, threshold and operator, its worst value, whether it passed, and the time of the sample. An inconclusive metric has passed: false, no value, and a message with its cause:

Controller metrics¶

The controller exports these metrics on its metrics endpoint:

When a rollback finishes, OME records a Normal event with the reason RolloutRunClosed and the message run <run ID> closed: RolledBack. ome_canary_rollback_total tells analysis and manual rollbacks apart.

Where the metrics come from¶

The canary's prometheus block picks the source:

Controller settings¶

The canaryAnalysis key of the inferenceservice-config ConfigMap sets the defaults that analysis uses. With the ome-resources chart, you set them under ome.controller.canaryAnalysis:

A Helm upgrade that changes these settings restarts the controller, so they all apply. After a kubectl edit of the ConfigMap, the last column applies, where 30 seconds is the default of ome.controller.configCacheTTL. A queryTimeout or cacheTTL that isn't a positive duration, or a maxConcurrency below 1, falls back to its default.

If you run without the bundled Prometheus, set bundledPrometheusAddress or defaultProvider, or canaries that name no source get only inconclusive samples. The chart fills in the bundled address even with prometheus.enabled: false, and the kustomize manifests in config/ leave it empty.

Metric provider bindings¶

The top-level metricProviders key of the same ConfigMap binds provider names to endpoints, for canaries and for the triggers of the alpha AutoscalerPolicy.

With the ome-resources chart, you set the key under ome.metricProviders:

ome:
  metricProviders:
    cluster-prometheus:
      serverAddress: "http://prometheus.monitoring.svc:9090"
      authSecretRef:
        name: prometheus-bearer
        key: token
      headers:
        X-Scope-OrgID: tenant-a

The older ome.autoscalerPolicy.metricProviders value counts only while the top-level key is absent.

When a provider isn't bound¶

A parked rollout sets the RolloutPlanReady condition to False, with the reason ProviderUnbound and the message groups[<N>]: metric provider "<name>" is not bound in this cluster's metricProviders configuration. OME also records a RolloutPlanParked Warning event.

What admission rejects¶

The API server enforces the required fields, the allowed values and the counts in Fields. It also rejects a prometheus block that sets both providerRef and serverAddress, with at most one of providerRef and serverAddress may be set. The OME webhook rejects these, where <where> is spec.rollout.groups[<N>].canary for an InferenceService and spec.canary for a RolloutPolicy:

A RolloutPolicy must name its metrics source through a binding, so its webhook also rejects these:

OME checks a policy's canary again when the rollout starts, and parks the rollout if it fails, with the reason PolicyNotReady or PlanInvalid: see When a rollout parks.

Related pages¶

• Canary progression: steps, capacity, traffic weights and the other gates.
• Promote or roll back a canary: override an analysis gate, or roll back.
• kubectl ome rollout: the rollout commands.
• Rollout policy: shared rollout settings, in alpha.
• Configure the controller: change the inferenceservice-config ConfigMap.
• Collect metrics: the bundled Prometheus, and scraping the controller.
• RolloutAnalysis and AnalysisPrometheus: the API reference.