Change the Instance status encoding Unreleased: coming in v1.3

OMENative keeps the status of each Instance on its component's InferenceReplica, as compact ColumnarV2 columns or a DenseV1 list. The chart sets ColumnarV2, so a new installation needs no change. Switch to DenseV1 when your own tools read the list. You set one Helm value, and the manager converts the InferenceReplicas after it restarts. The same steps switch back and change the bound, maxDecodedInstances.

Choose an encoding¶

Under ColumnarV2, OME writes the list instead when it's no larger than the columns, or when the InferenceReplica has more rows than maxDecodedInstances. A ColumnarV2 cluster therefore holds both encodings, and that's normal.

A script that reads status.instanceStatuses sees no rows on an InferenceReplica stored as columns, while kubectl ome instance reads both encodings. The guarded kubectl-ome actions, scale and migration start among them, decode both too, so their safety checks see lifecycle work that the columns store.

The omenativeStatus settings¶

Set them under ome.controller.omenativeStatus in your values file. The chart checks them, and writes them to the omenativeStatus key of the inferenceservice-config ConfigMap.

The manager reads the block only when it starts. It won't start when the block is missing or invalid, or when the cluster's InferenceReplica CRD is missing or too old. Upgrade the ome-crd chart before ome-resources. If you manage the ConfigMap yourself, add the block before you upgrade from v1.2.2.

Size maxDecodedInstances¶

Keep the chart's 20000 unless one InferenceReplica can hold more rows, counting the extra Instances of a surge, migration or replacement, and its Failed and Deleting rows. A larger one is stored as a list, where the compact form would help most.

OME stops managing a ColumnarV2 InferenceReplica with more rows than the bound. Never lower the bound below largestColumnarV2Rows in the preflight report. kubectl ome instance reads ColumnarV2 InferenceReplicas of up to 20,000 rows, whatever the bound. The guarded kubectl-ome actions decode at most 2048 rows for each InferenceReplica, also whatever the bound, so one that kubectl ome instance still reads can be too large to act on.

Step 1: Set the encoding¶

Check the block that the cluster runs now:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.omenativeStatus}{"\n"}'

On a default installation, it prints:

{
  "instanceStatusEncoding": "ColumnarV2",
  "maxDecodedInstances": 20000
}

Set the new block in your values file. This example moves to DenseV1. To move back, set ColumnarV2.

ome:
  controller:
    omenativeStatus:
      instanceStatusEncoding: DenseV1
      maxDecodedInstances: 20000

Keep your other values in the file, because helm upgrade resets any value that the file leaves out. Upgrade with the chart version that you run, from the CHART column of helm list -n ome, such as ome-resources-1.3.0:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

The upgrade restarts the manager, which reads the new block. To use kubectl instead, edit the ConfigMap and restart the manager, as in Change ConfigMap settings. Set the same block in your values file too, so that the next upgrade keeps it.

Wait for the new pods:

kubectl rollout status deployment ome-controller-manager -n ome

When they're running, the command ends with:

deployment "ome-controller-manager" successfully rolled out

Check the block that the replicas loaded:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 \
  | grep 'Configured InferenceReplica status representation'

When a replica starts, it logs the instanceStatusEncoding and maxDecodedInstances that it loaded, here DenseV1 and 20000. A bound of 0 means that none is set. A manager that rejects the block exits instead: see The manager doesn't start.

Step 2: Watch the InferenceReplicas convert¶

Most InferenceReplicas convert soon after the restart, depending on how many you have and on the manager's throughput settings. A conversion changes only how the rows are stored: the Instances and their pods carry on.

List the encodings:

kubectl get inferencereplicas -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,ENCODING:.status.instanceStatusEncoding'

ENCODING shows ColumnarV2 for columns, and <none> for a list. A move to DenseV1 is done when all rows show <none>:

NAMESPACE   NAME             ENCODING
serving     example-engine   <none>

Point list-reading tools at the InferenceReplicas only then. After a move to ColumnarV2, some rows keep <none> by design. An InferenceReplica that stays ColumnarV2 under DenseV1 is stuck: its status can't be decoded, or is too large for a list.

A conversion also records an InstanceStatusConverted event on the InferenceReplica:

kubectl get events -A --field-selector reason=InstanceStatusConverted

Its message names the InferenceReplica and says per-Instance status was rewritten from ColumnarV2 to DenseV1.

If Prometheus scrapes the manager, as Collect metrics sets up, these series track the change. Only the leader records them, and the encoding labels are dense_v1 or columnar_v2.

Check a fleet with ome-status-preflight¶

ome-status-preflight checks one or more clusters before and after a switch, and gives each a GO or NO-GO verdict. The check only reads. For each cluster, it needs a kubeconfig whose context can read the manager's Deployment and ConfigMap, list InferenceReplicas in all namespaces, and read OpenAPI v3 discovery.

Build it into bin/ome-status-preflight with Go 1.26 or later, from an OME checkout at the release that your manager runs:

make ome-status-preflight

The tool reads an inventory, a YAML file with no defaults that says what the clusters should run and how to reach them. Find the manager image for it:

kubectl get deployment ome-controller-manager -n ome \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="manager")].image}{"\n"}'

ghcr.io/moirai-internal/ome-manager:v1.3.0

Set omenativeStatus to the block that the clusters should run: the current one before a switch, and the new one after it. This inventory checks the move to DenseV1:

managerImage: ghcr.io/moirai-internal/ome-manager:v1.3.0
omenativeStatus:
  instanceStatusEncoding: DenseV1
  maxDecodedInstances: 20000
manager:
  namespace: ome
  deployment: ome-controller-manager
  container: manager
  configMap: inferenceservice-config
pageSize: 500
clusters:
  - name: prod-east
    kubeconfig: /etc/ome/kubeconfigs/prod-east
    context: prod-east-admin
  - name: prod-west
    kubeconfig: /etc/ome/kubeconfigs/prod-west
    context: prod-west-admin

Run the check:

bin/ome-status-preflight --inventory fleet.yaml

The report gives each cluster a verdict, with a - <reason>: <detail> line per problem, and counts its InferenceReplicas by encoding. When all clusters pass, it ends with:

result: GO (2 clusters)

The command exits with 0 on GO, 1 on NO-GO, and 2 on a bad flag or inventory. Add --json for a machine-readable report. The tool checks the ConfigMap. The log line in Step 1 shows what the running managers loaded.

Troubleshooting¶

helm upgrade fails on the omenativeStatus values¶

Helm rejects a bad block before it changes the cluster, with one of these messages:

• ome.controller.omenativeStatus.instanceStatusEncoding must be DenseV1 or ColumnarV2, got "<value>"
• ome.controller.omenativeStatus.maxDecodedInstances must be a positive integer, got <value>
• ome.controller.omenativeStatus.maxDecodedInstances is required when instanceStatusEncoding is ColumnarV2

Fix the value, and upgrade again.

The manager doesn't start¶

The new manager pods restart over and over, and kubectl rollout status keeps waiting. Find the error:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 \
  | grep -E 'Failed to initialize OMENative status configuration|status schema preflight failed'

The status can't be decoded¶

OME stops managing an InferenceReplica whose stored status it can't decode, and records a Warning event:

kubectl get events -A --field-selector reason=InstanceStatusDecodeFailed

The message names the InferenceReplica and the reason, in per-Instance status cannot be decoded (<reason>):

• cardinality_limit: the object is stored as ColumnarV2 with more rows than maxDecodedInstances, or the block sets no bound. Raise or restore the bound, as in Step 1.
• Any other reason, such as coverage or range_syntax: the stored status is damaged. Repair it.

A damaged status also refuses the guarded kubectl-ome actions that read the InferenceReplica, scale and migration start among them: they decode it themselves, under their own 2048-row bound in place of maxDecodedInstances. Instance status decoding lists the actions and their messages.

Status writes are rejected as too large¶

When the API server rejects an InferenceReplica's status write as too large, OME records a StatusSizeExceeded Warning event, and the InferenceReplica stalls until its status fits.

kubectl get events -A --field-selector reason=StatusSizeExceeded

The message gives the write's size and encoding, such as status write of <n> bytes (DenseV1).

• On DenseV1, switch to ColumnarV2 as in Step 1, with a bound that covers the InferenceReplica's rows.
• During a move to DenseV1, an InferenceReplica may be too large to store as a list, and it stays ColumnarV2. Set the target back to ColumnarV2.

Repair a damaged status¶

ome-status-preflight repair replaces one InferenceReplica's stored rows with rows that you write, and keeps the rest of its status. Use it only for a damaged status, never for cardinality_limit. It uses the inventory from the fleet check, and the cluster's context must be allowed to update inferencereplicas/status.

Write a row for every Instance, as a list under instanceStatuses and nothing else. An Instance's pods carry its index, incarnation and revision hash as labels, and a revision is named <InferenceService>-<component>-<hash>:

instanceStatuses:
- index: 0
  incarnation: 1
  phase: Ready
  runningRevision: example-engine-2f32f6fe
  podCount: 1
  servingPodCount: 1
  availablePodCount: 1
  admitted: true

Run the repair without --apply first. This example repairs example-engine, the engine of the InferenceService example:

bin/ome-status-preflight repair --inventory fleet.yaml --cluster prod-east \
  --namespace serving --name example-engine --replacement rows.yaml

The dry run checks your rows, and the cluster's block against the inventory's, and writes nothing. Its report starts with InferenceReplica status repair (dry run: nothing was written) and ends with Re-run with --apply to write.

If the report looks right, write it:

bin/ome-status-preflight repair --inventory fleet.yaml --cluster prod-east \
  --namespace serving --name example-engine --replacement rows.yaml --apply

The report starts with InferenceReplica status repair (applied). If the object changed after the tool read it, the tool refuses the write: run the dry run again. A refusal writes nothing, prints a refused: message, and exits with 1. Other errors print error: and exit with 2.

Next steps¶

• Read what a row holds in Where Instance status lives.
• Scrape the manager's metrics with Collect metrics.
• Look up every status field of an InferenceReplica in the API reference.