kubectl ome traffic Unreleased: coming in v1.3

kubectl ome traffic shows what the controller reports about how an InferenceService receives traffic, and drains workload clusters out of multi-cluster routing. The read commands report the traffic policy that OME wrote for your gateway and whether the gateway accepted it, the routes and endpoints in the status, and each canary's progress with the traffic weights that OME records for its revisions. explain compares that status with the traffic behavior you declared. The alpha actions add and remove drains, which ask multi-cluster routing to hold a workload cluster at weight 0 in the service's TrafficMap. Multi-cluster routing is alpha and still in development.

kubectl ome traffic SUBCOMMAND INFERENCESERVICE [flags]

Each subcommand takes the name of one InferenceService. The CLI looks for it in the namespace you give with -n, then in the namespace of your kubeconfig context, then in default.

status and explain read only the InferenceService. They don't read the policy that OME wrote, HTTPRoutes, Services, endpoints or pods. drain and undrain read the InferenceService and patch its annotations. A server dry run sends the patch too, so it needs the same permission. The kubectl-ome overview lists the RBAC each command needs, and Guarded actions the rule the actions add.

Each subcommand prints a table. For status and explain, -o wide prints more detail. For drain and undrain, it adds the target's UID and resourceVersion to the result. -o json and -o yaml print the whole report, with apiVersion: cli.ome.io/v1alpha1 and the kind TrafficStatusReport, TrafficExplainReport or ActionResult. On a terminal narrower than the table, the CLI wraps long cells, or prints each row as a list of fields when even the headers don't fit. Output to a file or pipe keeps one line per row.

The SOURCE column of status and explain says how the CLI got each value, as <evidence>/<freshness>, or - for a value that has neither, such as the report's kind. The evidence level is one of:

The freshness says whether the value is tied to the current generation of the InferenceService:

status¶

kubectl ome traffic status INFERENCESERVICE [flags]

status shows the traffic status that the controller wrote on the InferenceService: the policy that OME wrote for your gateway and whether the gateway accepted it, the load-balancing algorithm, the HTTPRoutes and endpoints, each canary's progress, and the traffic weight that OME records for each revision. It reads the InferenceService once and nothing else, so the report shows what the controller recorded, not proof that requests reach your pods. It doesn't show the TrafficMap or drains: drain and undrain say where to read them. kubectl ome status shows the same STATE, translator, algorithm and policy readiness in its Traffic row.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has the columns FIELD, COMP, VALUE and SOURCE. COMP names the component a row is about, or shows - for the whole service. The first five rows sum up the service:

STATE is the first of these that applies:

Pending and Reported become Partial when part of the evidence is missing or cut: UNSUPPORTED is Present, a condition is stale, or the report cut a list.

A service that declares no traffic behavior has no status.traffic, as Defaults when you declare nothing describes. For it, STATE is Unavailable and ISSUE lists TrafficStatusMissing, but the endpoints, canaries and revision weights from the rest of the status still appear.

The rows after the summary list what the status holds:

The role in a WEIGHT row says which revision the weight is for. In the component a CANARY row names, canary is the new revision and stable the one it replaces, and once the canary has passed its last step, the new revision is stable. In other components, stable is the revision that finished rolling out, from latestRolledoutRevision, or the only revision when that's unset and the revision is marked latest. Any other revision is other.

-o wide prints the five summary rows, then every value in full in place of the counts and WEIGHT rows:

The report keeps at most 4 routes, 16 endpoints and 8 revision weights in each component, the first in alphabetical order, and lists an issue when it cuts one of these lists. An issue marked invalid makes STATE Invalid:

-o json and -o yaml print a TrafficStatusReport. Its sources lists the InferenceService the CLI read, with its UID and generation, and content holds every value with its source. warnings lists PartialData when the report has an invalid issue or partial evidence, StaleEvidence when a condition is stale, and Truncated when the report cut a list. It's empty when none applies. The tables don't show warnings.

Examples¶

chat in prod declares RoundRobin behind Envoy Gateway, and its engine is at the first of two canary steps. OME has written the policy, and the gateway hasn't accepted or rejected it yet:

kubectl ome traffic status chat -n prod

FIELD          COMP     VALUE                 SOURCE
STATE          -        Pending               Computed/Unverifiable
TRANSLATOR     -        envoy-gateway         Computed/Current
ALGORITHM      -        RoundRobin            Reported/Current
POLICY-READY   -        Unknown/Pending       Reported/Current
UNSUPPORTED    -        None                  Reported/Current
ROUTES         -        2                     Reported/Current
ENDPOINTS      -        2                     Reported/Unverifiable
CANARY         engine   1/2 @ 20%             Reported/Unverifiable
WEIGHT         engine   stable:a1b2c3d4=80%   Reported/Unverifiable
WEIGHT         engine   canary:e5f6a7b8=20%   Reported/Unverifiable

Show every value behind those counts:

kubectl ome traffic status chat -n prod -o wide

FIELD          COMP     VALUE                                                              SOURCE
STATE          -        Pending                                                            Computed/Unverifiable
TRANSLATOR     -        envoy-gateway                                                      Computed/Current
ALGORITHM      -        RoundRobin                                                         Reported/Current
POLICY-READY   -        Unknown/Pending                                                    Reported/Current
UNSUPPORTED    -        None                                                               Reported/Current
POLICY         -        gateway.envoyproxy.io/v1alpha1/BackendTrafficPolicy/prod/chat      Reported/Current
ROUTE          -        chat                                                               Reported/Current
ROUTE          -        chat-engine                                                        Reported/Current
ENDPOINT       -        http://chat-engine.prod.svc.cluster.local                          Reported/Unverifiable
ENDPOINT       -        https://chat.prod.example/                                         Reported/Unverifiable
CANARY         engine   1/2 @ 20%                                                          Reported/Unverifiable
TARGET         engine   stable:chat-engine-rev-a1b2c3d4=80%                                Reported/Unverifiable
TARGET         engine   canary:chat-engine-rev-e5f6a7b8=20%                                Reported/Unverifiable
CONDITION      -        BackendPolicyReady=Unknown/Pending gen=7 at=2026-09-14T16:59:00Z   Reported/Current

With -o json, the same report is a TrafficStatusReport whose warnings is empty, since nothing in it is invalid, stale or cut.

status fails before it reads anything with unsupported output format "<value>" (supported: table, wide, json, yaml) for another -o value, and with invalid InferenceService name "<name>": and the reason for a name that isn't valid. When the read fails, it prints get InferenceService "<namespace>/<name>": followed by the API server's error, or, when the cluster doesn't serve the ome.io API, by OME does not appear to be installed on this cluster and a link to OME's README.

explain¶

kubectl ome traffic explain INFERENCESERVICE [flags]

explain compares the traffic behavior the InferenceService declares, in spec.traffic, its traffic annotations and its canary rollout groups, with the status the controller reports. It says whether the gateway supports the declared behavior, whether the status reports it realized, and where the two disagree. Like status, it reads the InferenceService once and nothing else, so a match means the controller's status agrees with the spec, not that requests follow it. The report never prints annotation values, header names, condition messages, credentials, UIDs or resourceVersions.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table has the columns LAYER, STATE, VALUE and SOURCE, with a row for each layer of the comparison:

SUMMARY is the first of these that applies:

INTENT is one of:

SUPPORT is the first of these that applies:

REALIZE is Invalid when the status has one of the issues RouteInvalid, EndpointInvalid, CanaryInvalid, AllocationInvalid, AllocationConflict and UnknownComponentStatus. Otherwise it's Unavailable when the status lists no routes, endpoints, revision weights or canaries, Partial when the report cut a list, and Reported when it lists some.

The checks compare the declared behavior with the status, not with requests:

algorithm and policy are NotApplicable when the service declares no traffic behavior for the gateway, and canary-weight when it has no canary rollout group. A check is Invalid when INTENT is Invalid or the evidence it needs is: canary-weight is Invalid, never Mismatch, when the status has CanaryInvalid or an allocation issue in a canary's component.

An ISSUE row lists each of these that applies, in this order:

-o wide prints a row for each value behind the verdict, with a FIELD column after LAYER:

-o json and -o yaml print a TrafficExplainReport. Its content holds the summary, the declared intent, the whole status report content under reported, the checks under comparisons, and the issues. Its sources lists the InferenceService without its UID. warnings repeats the warnings of the status report, and adds SourceUnavailable when the service declares traffic behavior and the status has no traffic section, and PartialData when SUMMARY is Partial or Unsupported.

Examples¶

chat in prod declares RoundRobin behind Envoy Gateway. The gateway has accepted the policy that OME wrote, and the status lists one HTTPRoute:

kubectl ome traffic explain chat -n prod

LAYER       STATE           VALUE           SOURCE
SUMMARY     Consistent      -               Computed/Current
INTENT      Declared        RoundRobin      Declared/Current
SUPPORT     Honored         -               Computed/Current
TRANSLATE   Computed        envoy-gateway   Computed/Current
REALIZE     Reported        r=1 e=0 w=0     Reported/Current
CHECK       Match           algorithm       Computed/Current
CHECK       Match           policy          Computed/Current
CHECK       NotApplicable   canary-weight   Computed/Current

On a cluster where the controller found no backend policy CRD when it started, OME writes no policy for the same service. With a status that lists no routes, endpoints or revision weights, explain shows that the gateway can't support the declared behavior and nothing is realized:

kubectl ome traffic explain chat -n prod

LAYER       STATE           VALUE                    SOURCE
SUMMARY     Unsupported     -                        Computed/Unavailable
INTENT      Declared        RoundRobin               Declared/Current
SUPPORT     Rejected        -                        Computed/Current
TRANSLATE   Computed        Unavailable              Computed/Current
REALIZE     Unavailable     r=0 e=0 w=0              Unavailable/Unavailable
CHECK       Match           algorithm                Computed/Current
CHECK       Mismatch        policy                   Computed/Current
CHECK       NotApplicable   canary-weight            Computed/Current
ISSUE       -               NoTranslatorAvailable    Computed/Unverifiable
ISSUE       -               RealizationUnavailable   Computed/Unverifiable

Show every value behind the first verdict:

kubectl ome traffic explain chat -n prod -o wide

LAYER      FIELD               STATE           VALUE                                                                     SOURCE
REPORT     api-version         -               cli.ome.io/v1alpha1                                                       -
REPORT     kind                -               TrafficExplainReport                                                      -
SUBJECT    namespace           -               prod                                                                      -
SUBJECT    name                -               chat                                                                      -
REPORT     collected-at        -               2026-09-14T17:00:00Z                                                      -
SOURCE     InferenceService    Reported        prod/chat gen=7 at=2026-09-14T17:00:00Z                                   Reported/Current
SUMMARY    state               Consistent      -                                                                         Computed/Current
SUMMARY    intent              Declared        -                                                                         Declared/Current
SUMMARY    support             Honored         -                                                                         Computed/Current
SUMMARY    realization         Reported        routes=1 endpoints=0 targets=0                                            Reported/Current
INTENT     algorithm           Declared        RoundRobin                                                                Declared/Current
INTENT     consistent-hash     Absent          -                                                                         Declared/Current
INTENT     endpoint-override   Absent          -                                                                         Declared/Current
REPORTED   algorithm           Reported        RoundRobin                                                                Reported/Current
REPORTED   translator          Computed        envoy-gateway                                                             Computed/Current
REPORTED   policy-ready        True            AcceptedByGateway                                                         Reported/Current
REPORTED   unsupported         None            -                                                                         Reported/Current
REPORTED   policy              Reported        gateway.envoyproxy.io/v1alpha1/BackendTrafficPolicy/prod/chat             Reported/Current
REPORTED   route               Reported        chat                                                                      Reported/Current
REPORTED   condition           Reported        BackendPolicyReady=True/AcceptedByGateway gen=7 at=2026-09-14T16:59:00Z   Reported/Current
CHECK      algorithm           Match           -                                                                         Computed/Current
CHECK      policy              Match           -                                                                         Computed/Current
CHECK      canary-weight       NotApplicable   -                                                                         Computed/Current

explain fails before it reads anything for the same -o values and names as status. When the read fails, it prints get InferenceService "<namespace>/<name>": and one of not found, forbidden, unauthorized, request cancelled, request timed out, request throttled, service unavailable, server error or API request failed, and never the API server's own text.

drain and undrain¶

kubectl ome traffic drain INFERENCESERVICE --workload-cluster CLUSTER --id ID --reason REASON [flags]
kubectl ome traffic undrain INFERENCESERVICE --id ID [flags]

drain asks multi-cluster routing to hold one workload cluster at weight 0 in the InferenceService's TrafficMap, for example during planned maintenance. It adds one entry to the ome.io/traffic-drain annotation on the control-plane InferenceService, keyed by --id, with the WorkloadCluster from --workload-cluster and your --reason. undrain removes one entry by its ID. For the drain in the examples, the annotation is:

metadata:
  annotations:
    ome.io/traffic-drain: '{"maintenance-a":{"cluster":"worker-a","reason":"planned maintenance"}}'

Each ID is added and removed on its own, so several drains can hold the same cluster, and the cluster gets its traffic back only when the last of them is removed. Adding or removing one ID keeps every other ID and every other annotation, and removing the last ID removes the annotation. The annotation belongs to the control-plane InferenceService: placement strips it from the copies that it makes on workload clusters, and both commands refuse a copy.

The routing controller reads the annotation each time it builds the TrafficMap. Each entry whose cluster matches a drain gets weight 0 and the drain's ID in drainRefs, and the other entries keep their weights, so the clusters that are left share the traffic in the same ratio as before. When drains hold every cluster that had a positive weight, the TrafficMap's Routable condition is False with the reason TrafficDrain. A drain changes only the TrafficMap: it doesn't mark the cluster unhealthy, and it doesn't scale, move or stop the replicas there. It's separate from allFailedPolicy: Drain in routing health probes. Why a cluster's weight is zero lists the other reasons an entry has weight 0.

The CLI doesn't check that multi-cluster routing is on, or that the cluster you name serves the InferenceService. Routing is off by default. It runs only on a control-plane cluster installed with the Helm values ome.multicluster.enabled=true, ome.multicluster.role=control-plane and ome.multicluster.config.routing.enabled=true, and skips a service that sets spec.routing.enabled: false. A drain whose cluster has no entry in the TrafficMap holds nothing, and when none of the service's drains matches an entry, the TrafficMap's OverrideActive condition is False with the reason OverridesPending. List the WorkloadClusters with kubectl get workloadclusters. kubectl ome cluster shows their status, and kubectl ome placement shows where the service is placed. --workload-cluster names the WorkloadCluster to drain. The global --cluster flag selects a cluster from your kubeconfig, and isn't an alias for it.

Nothing in OME removes a drain: it stays until you undrain it. To clear every drain at once, without the CLI's checks, remove the annotation with kubectl:

kubectl annotate inferenceservice chat -n prod ome.io/traffic-drain-

kubectl reports that it annotated the InferenceService, and the routing controller rebuilds the TrafficMap without drains. If a hand edit leaves the annotation malformed, both commands refuse, and the routing controller logs parse ome.io/traffic-drain: and keeps the last TrafficMap it built, rather than act as if there were no drains. Fix the value by hand, or remove the annotation.

Drain a workload cluster walks through a drain and an undrain.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

How the actions run¶

drain and undrain change the InferenceService in the same guarded way:

1. The CLI checks the flags and the name, reads the InferenceService, and refuses if the change isn't safe. A refusal changes nothing and exits 1.
2. It prints a preview of the exact change to stderr: the context, the namespace, the target with its UID and resourceVersion, the drain's ID, cluster and reason, the number of drains before and after, and the follow-up command. It prints the preview even with --yes.
3. It asks Confirm this exact action? [y/N] on stderr, and goes ahead only on y or yes, in any case. --yes confirms without the prompt, and without a terminal you must pass it.
4. It sends a JSON Patch that tests the UID and resourceVersion from the preview, then sets the ome.io/traffic-drain annotation to the new drains, or removes it when the last drain goes. If the InferenceService changed after the CLI read it, the API server rejects the patch and the CLI exits 3.
5. It prints the result, an ActionResult, to stdout.

--dry-run client runs every check, prints the preview and asks for confirmation, then stops without sending the patch. --dry-run server sends the same patch with dryRun=All, so the API server runs its checks and stores nothing.

The command has 45 seconds in all, including the time the prompt waits for your answer, and each API request has at most 10 seconds, or less if you set a shorter --request-timeout. When the time runs out, the command fails with context deadline exceeded. A credential plugin or custom transport in your kubeconfig may not stop when the time runs out. Without --yes, the command fails with action not confirmed; noninteractive input requires --yes when you answer anything but y or yes, type more than 16 characters, press Ctrl-D, or run it without a terminal. Ctrl-C at the prompt fails with context canceled. A declined or interrupted prompt sends no patch.

The ActionResult table shows:

The table cuts each value to 56 characters: a longer value keeps its first 53 characters and ends in .... So the client dry-run message prints as Validated locally; no patch sent. TrafficMap converge..., the accepted message as API accepted traffic annotation request; not TrafficM..., and the follow-up command prints cut once the name, namespace and context together pass 14 characters. -o json and -o yaml print every value in full, with the drain's ID, cluster and counts under traffic. The reason appears only in the preview and in the annotation.

applied means that the API server stored the annotation, and nothing more. A drain goes through three stages, and the command sees only the first:

1. The API server stores the annotation.
2. The routing controller rebuilds the TrafficMap with the drain applied.
3. The TrafficMap's publisher applies the new weights, and reports whether it has in the TrafficMap's Published condition. Publish a global endpoint describes what it writes.

To check the second stage, read the TrafficMap: the drained cluster's entry has weight 0 and the drain's ID in drainRefs, and OverrideActive is True with the reason OverridesApplied. Drain a workload cluster shows the commands. The follow-up command, traffic status, shows neither drains nor the TrafficMap, and kubectl ome wait has no condition for a drain. To list the drains on a service, read the annotation:

kubectl get inferenceservice chat -n prod -o jsonpath='{.metadata.annotations.ome\.io/traffic-drain}{"\n"}'

{"maintenance-a":{"cluster":"worker-a","reason":"planned maintenance"}}

If the patch fails with anything other than a refusal or a conflict, read the annotation before you run the command again, since the API server may have stored the patch. The error messages say to check traffic status, but traffic status doesn't show drains. The CLI never retries on its own. The error calls the outcome unknown when the API server's response is too large or doesn't match the request, as in API response is not bound to the request; outcome unknown, check traffic status. A server error, a timeout or a network error prints required Kubernetes API request failed; check access and connectivity or context deadline exceeded, even though the patch may have gone through. When the patch succeeds but the result can't be written, the command fails with write traffic action result failed; check traffic status. Each of these errors exits 1. Running a command again is safe: a drain that's already stored refuses with override ID already exists, and an undrain that already went through refuses with override ID does not exist.

Guarded actions describes the contract every mutating kubectl-ome command follows.

Refusals¶

Before it sends any request, each command checks its arguments in this order, and refuses on the first that fails:

The command then reads the InferenceService. When the read fails, it prints required Kubernetes API request failed; check access and connectivity, whether the InferenceService doesn't exist in the namespace, you lack get permission or the API server can't be reached, or context deadline exceeded when the read runs out of time. Check with kubectl get inferenceservice chat -n prod.

After the read, each command refuses in these cases, listed in the order the CLI first checks them:

Examples¶

The examples run against the control-plane cluster, whose kubeconfig context hub is the current context. Try a drain of worker-a for chat without storing it. The command prints its preview to stderr, as in the next example with Dry-run set to server, and then the result:

kubectl ome traffic drain chat -n prod \
  --workload-cluster worker-a \
  --id maintenance-a \
  --reason "planned maintenance" \
  --dry-run=server --yes

FIELD         VALUE
action        traffic drain
target        InferenceService/prod/chat
dry-run       server
accepted      Yes
applied       No
override-id   maintenance-a
cluster       worker-a
overrides     0 -> 1
message       API dry-run accepted; no changes persisted.
follow-up     kubectl ome traffic status chat -n prod --context=hub
hint          Use -o json or -o yaml for full values.

Drain worker-a, confirming at the prompt:

kubectl ome traffic drain chat -n prod --workload-cluster worker-a --id maintenance-a --reason "planned maintenance"

The command prints the preview and the prompt to stderr. Answer y:

ALPHA guarded traffic action (not TrafficMap convergence)
FIELD             VALUE
Action            traffic drain
Context           hub
Workload NS       prod
Target            InferenceService/chat
UID               2f6c1d9e-4b7a-4c1e-9f3d-8a5b6c7d0e12
ResourceVersion   1284711
Dry-run           none
Override ID       maintenance-a
Cluster           worker-a
Reason            "planned maintenance"
Override count    0 -> 1
Annotation        ome.io/traffic-drain
Follow-up         kubectl ome traffic status chat -n prod --context=hub
API acceptance is not TrafficMap convergence or data-plane realization.
The exact InferenceService UID/resourceVersion is tested atomically.
Other annotations and traffic-drain override IDs are preserved.
Run the previewed traffic status command to observe controller evidence.
Confirm this exact action? [y/N] y

Then it prints the result:

FIELD         VALUE
action        traffic drain
target        InferenceService/prod/chat
dry-run       none
accepted      Yes
applied       Yes
override-id   maintenance-a
cluster       worker-a
overrides     0 -> 1
message       API accepted traffic annotation request; not TrafficM...
follow-up     kubectl ome traffic status chat -n prod --context=hub
hint          Use -o json or -o yaml for full values.

Remove the drain without a prompt, as a script would, and print the result as JSON. After the preview, which shows Override count as 1 -> 0, the command prints:

kubectl ome traffic undrain chat -n prod --id maintenance-a --yes -o json

{
  "apiVersion": "cli.ome.io/v1alpha1",
  "kind": "ActionResult",
  "collectedAt": "2026-09-28T09:30:12.483920113Z",
  "action": "traffic undrain",
  "target": {
    "kind": "InferenceService",
    "namespace": "prod",
    "name": "chat",
    "uid": "2f6c1d9e-4b7a-4c1e-9f3d-8a5b6c7d0e12",
    "resourceVersion": "1284802"
  },
  "dryRun": "none",
  "accepted": true,
  "applied": true,
  "message": "API accepted traffic annotation request; not TrafficMap convergence.",
  "followUp": "kubectl ome traffic status chat -n prod --context=hub",
  "traffic": {
    "overrideID": "maintenance-a",
    "cluster": "worker-a",
    "overridesBefore": 1,
    "overridesAfter": 0
  }
}

Exit codes¶

Errors print to stderr as error: <message>.

Related guides¶

• Drain a workload cluster
• Troubleshoot an InferenceService