Provision TPU slices on demand Unreleased: coming in v1.3

Serve an OMENative component on GKE TPU node pools whose slices are provisioned on demand. You describe the pools once, in the operator's settings, and opt the component in with an annotation. For the pods of each Instance, OME then creates a Slice, the cluster-scoped GKE object that binds one TPU partition, waits for GKE to make it ready, creates the pods confined to it, and deletes the Slice once no pod needs it, which hands the capacity back.

You'll turn provisioning on for one accelerator, serve a single-pod engine on a four-chip slice, and watch OME create its Slice. An engine with a leader and workers takes the same steps with a multi-host topology: see Multi-host slices.

How OME provisions slices¶

OME provisions slices only for OMENative components, and only on node pools that carry the provision-only label. Every other pool holds static slices, which OME never provisions or releases: pods there schedule as usual, even when they opt in.

• One Slice serves the pods of one Instance. The leader and workers of a multi-pod Instance share it. A single-pod Instance gets one Slice per pod, so during an update's surge the new pod gets its own Slice while the pod it replaces keeps its own.
• The Slice's shape comes from what the pods already declare: the accelerator and topology node labels they select, and the TPU chips their containers request. The chips must fill the topology exactly: see Multi-host slices.
• OME creates an Instance's pods only once their Slice is ready: its Ready condition's reason is one of the configured readyStates. Each pod gets a node selector on the slice-name label, which confines it to the Slice's nodes. With slice.readyTimeout set, a Slice that has its partition but stays out of a ready state longer than the timeout is released and provisioned again, once no pod holds it.
• OME deletes a ready Slice that no pod holds when no Instance wants it anymore: after a scale-down, a shape change, or when one of its hosts can't take the pods. While pods of another workload hold TPU chips on the Slice's hosts, OME keeps it, because deleting it would deactivate the partition under them: see A SliceReleaseDeferred event appears. Deleting the Slice is what returns the capacity; GKE can hold it behind a finalizer while it releases the partition.
• When a Slice is deleted, or moved onto other nodes, from outside while pods run on it, OME deletes those pods so that their Instance is rebuilt: see A SliceLost event appears.

A Slice is named after the component's InferenceReplica, cut to fit, then the Instance's index, the pod's ordinal and an 8-character hash of the owner's identity, such as qwen3-tpu-engine-0-0-8f3a21c7. The ome.io/slice-owner-uid label alone makes a Slice OME's: OME never adopts, modifies or deletes a Slice without it. OME also labels its Slices with ome.io/managed-by: OMENative, the Instance's ome.io/instance-index, the pod's ome.io/pod-ordinal and the owner kind and name labels from Step 1, and annotates them with the owner as ome.io/slice-owner: {namespace}/{name}.

Step 1: Turn on slice provisioning¶

Describe the pools in the ome.controller.tpuSliceProvisioning Helm value. This example is the ome-resources chart's with a two-host topology and a ten-minute ready timeout added, for GKE dynamic slicing with single-host and two-host Ironwood slices:

ome:
  controller:
    tpuSliceProvisioning:
      chipResource: google.com/tpu
      nodeLabels:
        accelerator: cloud.google.com/gke-tpu-accelerator
        topology: cloud.google.com/gke-tpu-topology
        slice: cloud.google.com/gke-tpu-slice
      provisionOnly:
        key: cloud.google.com/gke-accelerator-topology-mode
        value: PROVISION_ONLY
      accelerators:
        tpu7x:
          sliceType: tpu7x
          chipsPerHost: 4
          topologies: ["2x2x1", "2x2x2"]
      slice:
        ownerKindLabel: cloud.google.com/slice-owner-kind
        ownerNameLabel: cloud.google.com/slice-owner-name
        annotations:
          cloud.google.com/managed-by: slice-scheduler
          slice.gke.io/retry-on-failure: "true"
        readyStates: [ACTIVE, ACTIVE_DEGRADED]
        readyTimeout: 10m

Upgrade the release with the chart version that you already run, keeping your other values in the file, as Configure the controller explains:

helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

The chart writes the block to the tpuSliceProvisioning entry of the inferenceservice-config ConfigMap, and the manager's pods roll, because the Deployment carries a checksum of the ConfigMap. Wait for the rollout:

kubectl rollout status deployment ome-controller-manager -n ome

deployment "ome-controller-manager" successfully rolled out

Check the entry:

kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.tpuSliceProvisioning}'

{"accelerators":{"tpu7x":{"chipsPerHost":4,"sliceType":"tpu7x","topologies":["2x2x1","2x2x2"]}},"chipResource":"google.com/tpu","nodeLabels":{"accelerator":"cloud.google.com/gke-tpu-accelerator","slice":"cloud.google.com/gke-tpu-slice","topology":"cloud.google.com/gke-tpu-topology"},"provisionOnly":{"key":"cloud.google.com/gke-accelerator-topology-mode","value":"PROVISION_ONLY"},"slice":{"annotations":{"cloud.google.com/managed-by":"slice-scheduler","slice.gke.io/retry-on-failure":"true"},"ownerKindLabel":"cloud.google.com/slice-owner-kind","ownerNameLabel":"cloud.google.com/slice-owner-name","readyStates":["ACTIVE","ACTIVE_DEGRADED"],"readyTimeout":"10m"}}

Step 2: Opt the engine in¶

Create a namespace for the InferenceService:

kubectl create namespace qwen3-tpu

namespace/qwen3-tpu created

Save this InferenceService as qwen3-tpu-isvc.yaml:

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: qwen3-tpu
  namespace: qwen3-tpu
spec:
  deploymentMode: OMENative
  model:
    name: qwen3-0-6b
  runtime:
    name: srt-qwen3-tpu
  engine:
    minReplicas: 1
    maxReplicas: 1
    annotations:
      ome.io/tpu-slice-provisioning: "true"
    nodeSelector:
      cloud.google.com/gke-tpu-accelerator: tpu7x
      cloud.google.com/gke-tpu-topology: 2x2x1
    tolerations:
      - key: google.com/tpu
        operator: Exists
        effect: NoSchedule
    runner:
      name: ome-container
      resources:
        requests:
          google.com/tpu: 4
        limits:
          google.com/tpu: 4

• The annotation ome.io/tpu-slice-provisioning: "true", in the component's annotations, opts the component's pods in. It reaches their pod template, where OME reads it. Each component opts in on its own, and any other value leaves provisioning off.
• The nodeSelector declares the shape with the labels from Step 1: a 2x2x1 slice of tpu7x, which is 4 chips on one host.
• The runner requests the slice's 4 chips, so the pod fills it exactly. OME rejects a shape the pods can't fill: see Troubleshooting.
• GKE taints TPU nodes with google.com/tpu, so keep the toleration unless the runtime already carries one.
• The nodeSelector, the toleration and the chip requests can also live in the runtime's engineConfig; the InferenceService's values win.

Create the InferenceService:

kubectl apply -f qwen3-tpu-isvc.yaml

inferenceservice.ome.io/qwen3-tpu created

OME creates the InferenceReplica qwen3-tpu-engine for the engine. Its Instance has no pods yet, so READY is 0:

kubectl get inferencereplica qwen3-tpu-engine -n qwen3-tpu

NAME               COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
qwen3-tpu-engine   engine      1         1         0       0           15s

Step 3: Watch the Slice and the pod¶

OME created a Slice for Instance 0's pod. List OME's Slices with their provisioning state, the Ready condition's reason:

kubectl get slices.accelerator.gke.io -l ome.io/managed-by=OMENative \
  -o 'custom-columns=NAME:.metadata.name,TYPE:.spec.type,TOPOLOGY:.spec.topology,STATE:.status.conditions[?(@.type=="Ready")].reason'

NAME                            TYPE    TOPOLOGY   STATE
qwen3-tpu-engine-0-0-8f3a21c7   tpu7x   2x2x1      ACTIVE

Your hash differs, and STATE is <none> until GKE reports one. While the state isn't one of the configured readyStates, OME withholds the pod: the Instance stays in Creating, its operation holds on CapacityProvisioning, and its last failure records the hold, with a message that ends with what the Slice waits for, such as slice qwen3-tpu-engine-0-0-8f3a21c7 is waiting for partition assignment. In kubectl ome instance status qwen3-tpu 0 --component engine -n qwen3-tpu, the failure row names the hold, as reason=CapacityProvisioning — cut in the default table, so add -o wide. The op hold row doesn't: it reads waiting=Unknown refused=-, with an issue row OperationWaitingUnknown, because CapacityProvisioning isn't a token the CLI recognizes. The wait doesn't count against the Instance's readiness deadline.

Once the Slice is ready, OME creates the pod. Wait for the InferenceService:

kubectl wait --for=condition=Ready inferenceservice/qwen3-tpu -n qwen3-tpu --timeout=30m

inferenceservice.ome.io/qwen3-tpu condition met

The pod is confined to the Slice by a node selector on the slice-name label, which OME set for you:

kubectl get pod qwen3-tpu-engine-0-default-0 -n qwen3-tpu \
  -o jsonpath='{.spec.nodeSelector.cloud\.google\.com/gke-tpu-slice}'

qwen3-tpu-engine-0-0-8f3a21c7

Send the engine a request as in Serve a model on OMENative.

Multi-host slices¶

For a slice that spans hosts, give the engine a leader and a worker, as Serve a multi-node model shows, and declare the shape on both: the Instance's pods share one Slice. On tpu7x from Step 1, a 2x2x2 topology is 8 chips on two 4-chip hosts: select it on the leader and the worker, with worker.size: 1 and 4 chips requested in each pod.

The pods of one Instance must fill their topology exactly, because GKE grants a slice whole and a TPU session must request exactly the slice's topology. OME checks that:

• every pod of the Instance selects the same accelerator and topology, and the topology is in the accelerator's topologies;
• every pod requests the same chip count under chipResource, summed over its containers from the limit, or the request when a container has no limit;
• a pod's chips fit on one host: the count divides chipsPerHost;
• together, the pods request exactly the topology's chips.

By default, an update surges: the new pods come up on a Slice of their own while the old pods keep theirs, so an update briefly needs the capacity of one more slice per updating Instance.

Metrics¶

The manager reports provisioning on its metrics endpoint, labeled by slice_type and topology. The created, released, release-deferred, lost and duration series of every configured type and topology start at zero when the manager starts. Collect metrics shows how to scrape them.

Troubleshooting¶

The InferenceService records ComponentReconcileError¶

When a change gives an opted-in Instance pods that can't fill a provisionable slice, the InferenceReplica admission webhook rejects OME's write of the engine's InferenceReplica, and the denial lands on the InferenceService as a ComponentReconcileError Warning event. Read it:

kubectl get events -n qwen3-tpu --field-selector reason=ComponentReconcileError \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

The message ends with the webhook's reason, which starts with pods cannot be placed on a provisioned slice:

The webhook rejects only a write that introduces the problem, and admits the InferenceReplica when it can't read the nodes. The controller then repeats the check before it creates pods: it withholds them and records a SliceDemandInvalid Warning event on the InferenceService, with the same reason.

Pods wait for a slice¶

The Instance stays in Creating with no pods, and its operation holds on CapacityProvisioning: in kubectl ome instance status, the failure row reads reason=CapacityProvisioning and the op hold row waiting=Unknown, as in Step 3. The Instance's last failure says what the Slice is waiting for, and kubectl describe slices.accelerator.gke.io qwen3-tpu-engine-0-0-8f3a21c7 shows GKE's Ready condition with its message. The deadline clock is stopped either way, and what ends the wait depends on what the Slice waits for:

• A Slice waiting for its partition waits for capacity, which a new Slice would wait for too, so OME waits as long as it takes, with or without slice.readyTimeout: free capacity in the pool.
• A Slice that has its partition but doesn't reach a ready state is recycled after slice.readyTimeout, when you set it: see A SliceReadyTimeout event appears. Without it, OME waits as long as the Slice isn't ready: fix what GKE's Ready condition reports.

A SliceReadyTimeout event appears¶

With slice.readyTimeout set in Step 1's values, a Slice that got its partition but has stayed out of a ready state for longer than the timeout is recycled once no pod holds it: OME withholds the pods, records the Warning event on the InferenceService, deletes the Slice and provisions the slot again. The new Slice gets a new partition, so a one-off Slice stuck short of ready heals on its own. The wait is measured from the Ready condition's last transition, or from the Slice's creation while GKE reports no condition. The delete itself waits while pods of another workload hold chips on the Slice's hosts: see A SliceReleaseDeferred event appears. Read the event:

kubectl get events -n qwen3-tpu --field-selector reason=SliceReadyTimeout \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

OMENative component=engine instance=0 withheld: slice qwen3-tpu-engine-0-0-8f3a21c7 is ACTIVATING for 11m3s, longer than the 10m0s ready timeout; it is released and provisioned again once no pod holds it

A Slice still waiting for its partition is never timed out, however long it waits: a new Slice would wait for the same capacity. A Slice whose pods are already running is left alone. When every replacement runs past the timeout too, the recycling repeats until you fix what GKE's Ready condition reports.

A SliceHostUnavailable event appears¶

A ready Slice that no pod holds has a node the pods can't be scheduled on: cordoned, not ready, or unreachable, and the pods don't tolerate the matching taint. OME withholds the pods, records the Warning event on the InferenceService, deletes the Slice and provisions the slot again, so a one-off broken host heals on its own. The delete itself waits while pods of another workload hold chips on the Slice's hosts: see A SliceReleaseDeferred event appears. Read the event:

kubectl get events -n qwen3-tpu --field-selector reason=SliceHostUnavailable \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

OMENative component=engine instance=0 withheld: slice qwen3-tpu-engine-0-0-8f3a21c7 is ACTIVE, but node gke-tpu-pool-node-3 is cordoned; it is released and provisioned again once no pod holds it

A Slice whose pods are already running is left alone. When every replacement Slice comes up broken too, fix the node pool.

A SliceLost event appears¶

OME deleted an Instance's running pods, and the Warning event on the InferenceService says why: their Slice was deleted, or moved onto other nodes, from outside while they ran on it. OME releases a Slice only once no pod holds it and creates pods only once their Slice is ready, so pods can lose their Slice only through a change from outside, such as an administrator or a script deleting it. The pods' chips go with the partition, so OME deletes the pods and rebuilds the Instance as in Step 3: it provisions a new Slice when the old one is gone, waits until the slot's Slice is ready, and creates new pods confined to it. Read the event:

kubectl get events -n qwen3-tpu --field-selector reason=SliceLost \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

TPU slice qwen3-tpu-engine-0-0-8f3a21c7 was deleted while pods qwen3-tpu-engine-0-default-0 ran on it; they are deleted so that their Instance is rebuilt

The message names the pods OME deleted and says how the Slice was lost:

• was deleted: the Slice is gone, is being deleted, or was re-created from outside, which makes it younger than its pods.
• lost its partitions: the Slice exists but no longer binds a TPU partition.
• moved off node gke-tpu-pool-node-3: a node a pod is bound to no longer carries the Slice's slice-name label, or is gone. Every bound pod of the Instance is deleted then, those on nodes the Slice still holds too, because they started together on the hosts the Slice held before. A pod not yet bound isn't: its node selector follows the label to the Slice's new hosts.

OME confirms a lost Slice with a live read before it deletes anything, since its cache may lag, and deletes each pod under a UID precondition, so it never deletes a replacement pod. Each recovery counts on ome_tpu_slice_lost_total: see Metrics. The Instance heals on its own, but whatever took the Slice can take the replacement too: nothing but OME should delete the Slices it provisions, so find the deleter, starting from the cluster's audit logs for the Slice's name.

A SliceReleaseDeferred event appears¶

OME has a Slice to release, after a scale-down, a shape change, a recycle or a deletion, but pods of another workload hold TPU chips on its hosts, so it keeps the Slice: deleting it would deactivate the partition under those pods. OME records the Warning event on the component's InferenceReplica and retries on each reconcile, and it releases the Slice once the pods are gone. Read the event:

kubectl get events -n qwen3-tpu --field-selector reason=SliceReleaseDeferred \
  -o jsonpath='{range .items[*]}{.message}{"\n"}{end}'

TPU slice qwen3-tpu-engine-0-0-8f3a21c7 is kept: pods of other workloads hold chips on its hosts (tpu-batch/indexer-0); it is released once they are gone

A pod holds the Slice's hosts while it is bound to one of its nodes, requests chips under chipResource, isn't confined to this Slice by the slice-name node selector, and hasn't succeeded or failed. The message names each one as {namespace}/{name}: wait for them to finish, or remove them where they come from. Each deferred attempt counts on ome_tpu_slice_release_deferred_total: see Metrics.

A SliceOwnershipConflict event appears¶

An object that isn't the component's InferenceReplica holds a Slice at the name OME needs, so OME withholds the slot's pods and records the Warning event. OME never adopts, modifies or deletes a Slice without its owner's ome.io/slice-owner-uid label. See whose it is with kubectl get slices.accelerator.gke.io qwen3-tpu-engine-0-0-8f3a21c7 --show-labels, then rename the InferenceService or remove the conflicting Slice where it came from.

No Slices are created¶

The pods exist but stay Pending, or nothing happens at all, and kubectl get slices.accelerator.gke.io -l ome.io/managed-by=OMENative lists nothing. Check, in order:

1. The tpuSliceProvisioning entry is set: Step 1's ConfigMap check prints the block, not an empty result.

2. The manager found the Slice CRD when it started. A manager that didn't logs this, and provisions nothing until you restart it with kubectl rollout restart deployment ome-controller-manager -n ome:

   ShellCopy

   kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep "won't provision TPU slices"

   Output

   The InferenceReplica controller won't provision TPU slices because the accelerator.gke.io/v1beta1 Slice CRD is not available; pods are created without slices until it is installed and the manager restarts.

3. The component's annotations carry ome.io/tpu-slice-provisioning: "true", exactly.

4. A node with both the accelerator label's value and the provision-only label exists. Without one, OME treats the pool as static and creates the pods unconfined, which isn't an error: kubectl get nodes -l cloud.google.com/gke-tpu-accelerator=tpu7x,cloud.google.com/gke-accelerator-topology-mode=PROVISION_ONLY must list the pool's nodes.

The manager doesn't start after the upgrade¶

kubectl rollout status keeps waiting, and the new pods' logs show Failed to initialize TPU slice provisioning configuration, with every field that is missing or invalid. Fix the values and upgrade again. The manager rollout doesn't finish explains how the Deployment behaves meanwhile.

Clean up¶

Delete the InferenceService and its namespace. OME releases each Instance's Slices as part of removing it, so the deletion waits while pods of other workloads hold chips on their hosts (see A SliceReleaseDeferred event appears) and until GKE lets them go:

kubectl delete inferenceservice qwen3-tpu -n qwen3-tpu
kubectl delete namespace qwen3-tpu

inferenceservice.ome.io "qwen3-tpu" deleted from qwen3-tpu namespace
namespace "qwen3-tpu" deleted

Check that no Slice of the engine remains:

kubectl get slices.accelerator.gke.io -l cloud.google.com/slice-owner-name=qwen3-tpu-engine

No resources found

When a deleting InferenceReplica runs past its teardown deadline with a release still failing, OME lets the deletion finish anyway and records a Warning that ends with delete the slices labeled ome.io/slice-owner-uid={uid} by hand, with the InferenceReplica's UID. Delete them with that label selector.

The model and the runtime stay. To delete the model, follow Clean up in Serve your first model.

Next steps¶

• Serve a multi-node model: set up the leader and workers that share a multi-host slice.
• Set Instance readiness deadlines: bound how long an Instance may take once its slice is ready.
• Instances: the phases, operations and holds this guide reads.
• Collect metrics: scrape the manager's metrics, the slice series included.