kubectl ome placement Preview Unreleased: coming in v1.3

kubectl ome placement shows where an InferenceService runs across workload clusters. It shows the placement that the control plane reports, the registered WorkloadClusters and the routing table in its TrafficMap. The command reads multi-cluster placement and routing, which are alpha, still in development and off by default, so what it reports can change between releases. Turn on routing shows the Helm values for the control-plane cluster. Run the command with the control plane as your current context. It reads what the control plane recorded and changes nothing. kubectl ome cluster shows the workload clusters themselves.

kubectl ome placement SUBCOMMAND INFERENCESERVICE [flags]

Each subcommand prints a FIELD and VALUE table, one row per fact. The table cuts long field names and values and ends them with ..., so Selector compat... is the Selector compatible row. The -o wide output adds rows, and -o json and -o yaml print the whole report, uncut, as Output formats describes. The problems that the CLI finds are in the Issue rows and in content.issues, and warnings is always empty. Required RBAC lists the permissions that each subcommand needs.

The CLI calls a workload cluster in the InferenceService's status.placement.candidates a home, and an entry of its TrafficMap a route home. These rules hold in all three tables:

• An address shows only its origin: its scheme, host and port. It reads NotRecorded when it's missing, and InvalidEndpoint when the URL is malformed or uses a scheme other than http or https. In JSON, pathPresent says whether the URL had a path.
• An unrecognized mode or phase reads Unknown, and an unrecognized reason reads OtherReportedReason.
• Homes, clusters and route homes are sorted by name. With more than four, the table shows a preview row such as Home preview 4/6; use -o wide or json, then the first four. -o wide shows the rest.
• The tables end with Hint rows and a View row, Bounded cells; use -o json for complete identities.

Freshness values say whether a row describes the current version of its object. The CLI compares the generation that the controller recorded with the object's metadata.generation:

Rows named inspection or inspect show how the CLI read a list, as <state> <kept>/<total>; truncated=<bool>, where the report keeps <kept> of the <total> items:

status¶

kubectl ome placement status INFERENCESERVICE [flags]

status shows the placement that the controller reports in the InferenceService's status.placement: its phase, and each home with its endpoint. It also shows the placement inputs that the InferenceService declares. OME places an InferenceService that sets spec.placement, which requires policy: ClusterAffinity and a mode; its optional clusterAffinity terms restrict which WorkloadClusters match, and without them every registered cluster matches. OME rejects the legacy ome.io/accelerator-requirements and ome.io/cluster-selector annotations on write and no longer places an InferenceService stored with one, but the reports still read such an object's annotations as its declared selectors. Placement reads NotRecorded for an InferenceService that OME hasn't tried to place, and on a cluster that isn't a control plane. When Placement reads Pending, explain shows the registered clusters and whether they're Ready.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

All three subcommands start with these rows:

The table has no row for a declared spec.placement.split: only the JSON and YAML reports record it, under inputs.split.

Then each home gets these rows:

-o wide adds these rows to each home:

The table ends with the hint Placed does not prove replica floor met: a cluster has admitted the InferenceService, but it may not have the replicas it needs yet.

Examples¶

On a control plane, show the placement of chat in prod. It sets spec.placement with policy: ClusterAffinity, mode: All and a clusterAffinity term that requires the label pool=gpu, and OME placed it on the two workload clusters labeled pool=gpu:

kubectl ome placement status chat -n prod

FIELD                VALUE
Mode                 All (Declared)
Placement            Placed (reported)
Freshness            Unverifiable
Input source         Structured
Selectors            NoRequirements
Reported cluster
Endpoint origin      NotRecorded
Service origin       NotRecorded
Home inspection      Validated 2/2; truncated=false
Provenance inspect   Validated 2/2; truncated=false
Reported home        worker-a (Admitted)
Home origin          https://chat.worker-a.example.com
Reported home        worker-b (Admitted)
Home origin          https://chat.worker-b.example.com
Hint                 Placed does not prove replica floor met
View                 Bounded cells; use -o json for complete identities

Selectors reads NoRequirements because chat declares its restriction in clusterAffinity, not in the legacy annotations.

With -o wide, each home also gets its replica counts and provenance. For worker-a, these rows follow its Home origin row:

kubectl ome placement status chat -n prod -o wide

Admitted replicas    2 (Reported)
Ready replicas       2 (Reported)
Provenance           NotRecorded; freshness unverifiable
Policy inspection    NotRecorded 0/0; truncated=false
Group inspection     NotRecorded 0/0; truncated=false

Provenance reads NotRecorded because chat references no AutoscalerPolicy or RolloutPolicy.

On a single cluster, for a chat that omits spec.placement, the same command shows that nothing placed it:

kubectl ome placement status chat -n prod

FIELD                VALUE
Mode                 Single (Defaulted)
Placement            NotRecorded (reported)
Freshness            Unverifiable
Input source         LegacyAnnotations
Selectors            NoRequirements
Reported cluster
Endpoint origin      NotRecorded
Service origin       http://chat.prod.svc.cluster.local:8080
Home inspection      NotRecorded 0/0; truncated=false
Provenance inspect   NotRecorded 0/0; truncated=false
Hint                 Placed does not prove replica floor met
View                 Bounded cells; use -o json for complete identities

The only address is the origin of its status.url.

explain¶

kubectl ome placement explain INFERENCESERVICE [flags]

explain lists each registered WorkloadCluster with its reported readiness, and sums up the routing settings in the InferenceService's spec.routing. For an InferenceService stored with the legacy annotations, it also checks their selectors against each cluster's labels. It doesn't evaluate clusterAffinity terms, so for an InferenceService that sets spec.placement, Selector compatible reads NotApplicable on every cluster. A match doesn't make a cluster eligible: explain doesn't say where the controller can place the InferenceService.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table starts with the same rows as status, without the homes. Then come the routing rows:

When Routing intent isn't Absent, four rows follow:

With -o wide, a Configured probe adds rows that sum up its settings: Probe method, Probe statuses, Probe thresholds, Probe timing and All-failed policy. A Configured capacity poll adds Capacity method, Capacity window and Capacity timing. A setting that isn't set reads -.

The Fleet rows say how the CLI listed the WorkloadClusters:

Fleet Returned, Fleet Admitted, Fleet Pages, Fleet Complete and Fleet Truncated describe what the CLI read.

Then each cluster gets these rows:

-o wide adds these rows to each cluster:

The table ends with two hints, WLC Ready is reachability, not capacity and Partial fleet: no global eligibility verdict. The second prints even when the CLI read the whole fleet.

Examples¶

On the control plane from the status examples, with a third workload cluster, worker-c, labeled pool=cpu:

kubectl ome placement explain chat -n prod

FIELD                VALUE
Mode                 All (Declared)
Placement            Placed (reported)
Freshness            Unverifiable
Input source         Structured
Selectors            NoRequirements
Reported cluster
Endpoint origin      NotRecorded
Service origin       NotRecorded
Home inspection      Validated 2/2; truncated=false
Provenance inspect   Validated 2/2; truncated=false
Routing intent       Absent
Routing enablement   Inherited
Install routing      Gate/defaults unobserved
Fleet state          Observed
Fleet reason
Fleet Returned       3
Fleet Admitted       3
Fleet Pages          1
Fleet Complete       true
Fleet Truncated      false
Cluster              worker-a
Selector compat...   NotApplicable
Reported WLC Ready   True (Current)
Condition inspect    Validated 1/1; truncated=false
Cluster              worker-b
Selector compat...   NotApplicable
Reported WLC Ready   True (Current)
Condition inspect    Validated 1/1; truncated=false
Cluster              worker-c
Selector compat...   NotApplicable
Reported WLC Ready   True (Current)
Condition inspect    Validated 1/1; truncated=false
Hint                 WLC Ready is reachability, not capacity
Hint                 Partial fleet: no global eligibility verdict
View                 Bounded cells; use -o json for complete identities

chat restricts placement with a clusterAffinity term, which explain doesn't evaluate, so Selector compatible reads NotApplicable on all three clusters, including worker-c, where OME didn't place it. chat sets no spec.routing, so Routing intent is Absent and the table leaves out the four routing detail rows.

With -o wide, these rows follow the Condition inspect row of worker-a:

kubectl ome placement explain chat -n prod -o wide

Reported home        True
Connection source    KubeConfigReferenceNotResolved
Ready reason         OtherReportedReason

worker-a connects through a kubeconfig Secret, and its Ready reason, Reachable, shows as OtherReportedReason. worker-c shows False for Reported home.

endpoint¶

kubectl ome placement endpoint INFERENCESERVICE [flags]

endpoint shows the routing table that the controller computed for the InferenceService, from the TrafficMap with the same name and namespace. It shows the TrafficMap's conditions, whether the publisher acknowledged it, and each route home with its endpoint, weight and recorded probe. These are recorded values: the CLI doesn't probe the endpoints or read the gateway. OME writes the TrafficMap only when routing is on, as Turn on routing shows. Traffic map describes the fields.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

Output fields¶

The table starts with the same rows as status, without the homes. The TrafficMap rows follow:

TrafficMap Returned, Admitted, Pages, Complete and Truncated then describe the read: 1, 1, 1, true and false when the report uses the TrafficMap. Then come the rows about its contents:

A condition that the TrafficMap doesn't have reads Unknown: NotRecorded (Unverifiable). Conditions lists each condition's reasons.

While a publish runs, Published reads False: OtherReportedReason and Publisher reads Invalid. Run the command again when the publish finishes. This is a known bug.

Then each route home gets these rows:

-o wide adds these rows to each route home. The first three appear only when the entry records its capacity:

The table ends with two hints, Routing generation does not date placement and Recorded probes only; no CLI network probe. The first means that Routing freshness says nothing about how current the placement status is.

Examples¶

On the control plane from the status examples, with routing on, while worker-a is drained for maintenance:

kubectl ome placement endpoint chat -n prod

FIELD                VALUE
Mode                 All (Declared)
Placement            Placed (reported)
Freshness            Unverifiable
Input source         Structured
Selectors            NoRequirements
Reported cluster
Endpoint origin      NotRecorded
Service origin       NotRecorded
Home inspection      Validated 2/2; truncated=false
Provenance inspect   Validated 2/2; truncated=false
TrafficMap state     Observed
TrafficMap reason
TrafficMap Retu...   1
TrafficMap Admi...   1
TrafficMap Pages     1
TrafficMap Comp...   true
TrafficMap Trun...   false
Entry inspection     Validated 2/2; truncated=false
Probe inspection     Validated 2/2; truncated=false
Condition inspect    Validated 4/4; truncated=false
Routing freshness    Current
Traffic override     True: OverridesApplied (Current)
Capacity fallback    False: CapacityPollingDisabled (Current)
Routable             True: Routable
Routable freshness   Current:
Published            True: Published (Current)
Publisher            ReportedTrue
Publisher fresh      Current
Route home           worker-a
Endpoint origin      https://chat.worker-a.example.com
Final weight         0; healthy=true
Recorded probe       NotRecorded
Route evidence       drains=1; gate=NotRecorded; fallback=NotRecorded
Probe policy         NotRecorded
Route home           worker-b
Endpoint origin      https://chat.worker-b.example.com
Final weight         1; healthy=true
Recorded probe       NotRecorded
Route evidence       drains=0; gate=NotRecorded; fallback=NotRecorded
Probe policy         NotRecorded
Hint                 Routing generation does not date placement
Hint                 Recorded probes only; no CLI network probe
View                 Bounded cells; use -o json for complete identities

One drain holds worker-a at weight 0, so worker-b gets all the traffic and OverrideActive is True. The TrafficMap reflects the current InferenceService, and the publisher has acknowledged its current generation. The entries record no probe results, so Recorded probe and Probe policy read NotRecorded.

With -o wide, these rows follow the Probe policy row of worker-a:

kubectl ome placement endpoint chat -n prod -o wide

Capacity proven...   ControlPlane
Allocated count      2 (Reported)
Ready count          2 (Reported)
Probe freshness      Unverifiable
Drain refs           maintenance-a
Probe gate           NotRecorded
Home cap fallback    NotRecorded

maintenance-a is the ID of the drain, added with the alpha kubectl ome traffic drain command. Drain a workload cluster shows how to add and withdraw one.

Exit codes¶

Errors print to stderr as error: <message>.

Related guides¶

• Configure routing health probes
• Weight traffic for heterogeneous clusters
• Tune routing capacity polling
• Troubleshoot an InferenceService