Drain a workload cluster Preview Unreleased: coming in v1.3

Drain a workload cluster to move an InferenceService's traffic off it, for example during maintenance, while its replicas there keep running. OME holds the drained cluster at weight 0 in the service's TrafficMap, and the other clusters take its share. Start a drain with kubectl ome traffic drain, and end it with kubectl ome traffic undrain. A drain applies to one InferenceService: to take a cluster out of every service's traffic, drain each service. Multi-cluster routing is alpha and still in development, and so are both commands. Routing is off by default.

How a drain is recorded¶

kubectl ome traffic drain adds an entry, keyed by a drain ID that you choose, to the ome.io/traffic-drain annotation on the InferenceService in the control-plane cluster:

metadata:
  annotations:
    ome.io/traffic-drain: '{"maintenance-a":{"cluster":"worker-a","reason":"planned maintenance"}}'

OME gives the drained cluster's TrafficMap entry weight 0, and lists the drain's ID in its drainRefs. The other clusters keep their weights, and share the traffic in the same ratio as before. Several drains can hold one cluster, and it gets its traffic back when the last one is removed. A drain stays until you undrain it.

Step 1: Drain the cluster¶

First check that another cluster can take the traffic:

kubectl get trafficmap chat -n prod \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{" weight="}{.weight}{" drainRefs="}{.drainRefs}{"\n"}{end}'

Both clusters have the same capacity, so each has weight 1:

worker-a weight=1 drainRefs=
worker-b weight=1 drainRefs=

kubectl ome traffic drain takes three required flags:

Drain worker-a. To try it first, add --dry-run=server: the API server checks the change and your permission, and stores nothing.

kubectl ome traffic drain chat -n prod \
  --workload-cluster worker-a \
  --id maintenance-a \
  --reason "planned maintenance"

The command prints a preview to stderr. Check that its Cluster row names one of the entries you listed, since the command accepts any valid name, then type y and press Enter:

ALPHA guarded traffic action (not TrafficMap convergence)
FIELD             VALUE
Action            traffic drain
Context           hub
Workload NS       prod
Target            InferenceService/chat
UID               2f6c1d9e-4b7a-4c1e-9f3d-8a5b6c7d0e12
ResourceVersion   1284711
Dry-run           none
Override ID       maintenance-a
Cluster           worker-a
Reason            "planned maintenance"
Override count    0 -> 1
Annotation        ome.io/traffic-drain
Follow-up         kubectl ome traffic status chat -n prod --context=hub
API acceptance is not TrafficMap convergence or data-plane realization.
The exact InferenceService UID/resourceVersion is tested atomically.
Other annotations and traffic-drain override IDs are preserved.
Run the previewed traffic status command to observe controller evidence.
Confirm this exact action? [y/N] y

In a script, pass --yes to skip the prompt. The result goes to stdout:

FIELD         VALUE
action        traffic drain
target        InferenceService/prod/chat
dry-run       none
accepted      Yes
applied       Yes
override-id   maintenance-a
cluster       worker-a
overrides     0 -> 1
message       API accepted traffic annotation request; not TrafficM...
follow-up     kubectl ome traffic status chat -n prod --context=hub
hint          Use -o json or -o yaml for full values.

Step 2: Check the drain¶

applied Yes means that the API server stored the drain, and OME then applies it to the TrafficMap. The result suggests kubectl ome traffic status, but that command doesn't show drains, so list the TrafficMap's entries:

kubectl get trafficmap chat -n prod \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{" weight="}{.weight}{" drainRefs="}{.drainRefs}{"\n"}{end}'

worker-a weight=0 drainRefs=["maintenance-a"]
worker-b weight=1 drainRefs=

worker-a is held at 0, and worker-b gets all the traffic. Then check the TrafficMap's conditions:

kubectl get trafficmap chat -n prod

NAME   MODE   PUBLISHED   ROUTABLE   OVERRIDE   REASON     AGE
chat   All    True        True       True       Routable   3h

OVERRIDE is True because the drain holds an entry, and ROUTABLE is True because worker-b can still serve. This PUBLISHED can still be True from before the drain. Before you start the maintenance, wait until kubectl ome get trafficmap chat -n prod shows PUBLISHED True. That command shows True only once the publisher has written the new weights to your gateway's route. If it stays Unknown or False, see Check the publication. Clients that call worker-a's own endpoint, not the global one, still reach it.

To see every drain on the service, with its reason, read the annotation:

kubectl get inferenceservice chat -n prod -o jsonpath='{.metadata.annotations.ome\.io/traffic-drain}{"\n"}'

{"maintenance-a":{"cluster":"worker-a","reason":"planned maintenance"}}

Step 3: Undrain the cluster¶

When the maintenance is done, remove the drain by its ID:

kubectl ome traffic undrain chat -n prod --id maintenance-a

Check the drain's cluster and reason in the preview, and type y. The result shows applied Yes and overrides 1 -> 0:

FIELD         VALUE
action        traffic undrain
target        InferenceService/prod/chat
dry-run       none
accepted      Yes
applied       Yes
override-id   maintenance-a
cluster       worker-a
overrides     1 -> 0
message       API accepted traffic annotation request; not TrafficM...
follow-up     kubectl ome traffic status chat -n prod --context=hub
hint          Use -o json or -o yaml for full values.

OME then sets worker-a's weight from its capacity and health again. If the maintenance left it with fewer ready replicas, its weight stays lower, or at 0, until they're ready: see Why a cluster's weight is zero. List the entries again:

kubectl get trafficmap chat -n prod \
  -o jsonpath='{range .spec.entries[*]}{.cluster}{" weight="}{.weight}{" drainRefs="}{.drainRefs}{"\n"}{end}'

worker-a weight=1 drainRefs=
worker-b weight=1 drainRefs=

Troubleshooting¶

The commands print errors to stderr as error: <message>. The reference lists their exit codes.

The command refuses to run¶

These are the messages you're most likely to see. Refusals lists the rest.

The patch is rejected as stale¶

The command fails with this message and exit code 3:

error: guarded annotation patch rejected; refresh traffic status and retry explicitly

chat changed after the command read it, often through a status update while the prompt waited, so the API server applied nothing. Run the command again to see a fresh preview.

The command fails after you confirm¶

When the command fails after you confirm, or after the preview when you pass --yes, the API server may still have stored the change. This applies to a message that ends in check traffic status, and to required Kubernetes API request failed; check access and connectivity or context deadline exceeded. To see whether it did, list the service's drains, as in Step 2. Running the command again is also safe. If the first one went through, a repeated drain fails with traffic drain refused: override ID already exists. A repeated undrain fails with traffic undrain refused: override ID does not exist.

The drain stays pending¶

In Step 2, no entry lists the drain in drainRefs. OVERRIDE is False too, unless another drain holds an entry. The drain names a cluster that has no entry in the TrafficMap, usually because of a typo. Undrain it, then drain the right cluster: until you do, it holds any cluster with that name that later serves the service.

Routable is False with reason TrafficDrain¶

The drains hold every cluster that had a positive weight, so the gateway has nowhere to send chat's requests. You drained them all, or the others were already at 0, for example with no ready replicas. Undrain a cluster that can serve, or fix the others.

The TrafficMap stops changing after a hand edit¶

Nothing validates a hand edit of ome.io/traffic-drain. When the value is invalid, OME keeps the last valid TrafficMap, so drained clusters stay drained. Capacity and health changes stop reaching the TrafficMap too, and kubectl ome traffic drain and undrain refuse with traffic action refused: existing traffic-drain annotation is malformed or unsafe. OME logs the error:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep 'parse ome.io/traffic-drain'

Each line holds the parser's error after parse ome.io/traffic-drain:. Fix the value with kubectl annotate --overwrite, in the form that How a drain is recorded shows, with IDs, clusters and reasons that kubectl ome traffic drain accepts. Removing the annotation also works, but it ends every drain on the service.

Next steps¶

• Traffic map: how OME sets each cluster's weight, and what the conditions mean.
• kubectl ome traffic: every flag of drain, undrain, status and explain.
• Guarded actions: the preview and confirmation that mutating kubectl ome commands share.