Move from LeaderWorkerSet to OMENative Unreleased: coming in v1.3

Move a multi-node component off the deprecated MultiNode deployment mode, which runs it on a LeaderWorkerSet, and onto OMENative, OME's own way of running an inference component. OMENative runs each replica as an Instance: a leader pod and its workers, which OME creates, updates, repairs, moves and removes as one unit. On OMENative:

• Updates bring a new Instance up and serving before the old one drains, by default. Rollout groups can roll out prefill and decode together, by canary, blue-green or rolling update.
• A revision that keeps failing is held until you change the component or release the revision.
• When one of an Instance's pods fails, OME rebuilds the whole Instance, by default.
• You can move an Instance off a node, and OME starts its replacement first. The alpha Alfred recommends moves and, when you let it, requests them. On MultiNode, it only advises, with the reason LWSMigrationUnsupported.
• The component reports per-Instance status, and can have an autoscaler and a PodDisruptionBudget.
• Gang scheduling works with the PodGroup CRD and a gang-aware scheduler, such as the alpha OME scheduler, which needs Kubernetes 1.35.

The steps move the engine of the InferenceService deepseek-r1, then delete its old LeaderWorkerSet.

What changes when a component moves¶

When a component's ome.io/deploymentMode annotation changes from MultiNode to OMENative, OME:

• Creates the InferenceReplica deepseek-r1-engine, which runs the component's Instances.
• Points the component's Service, deepseek-r1-engine, at the new pods at once, so the component serves nothing until they're ready.
• Leaves the LeaderWorkerSet lws-deepseek-r1-engine running until you delete it. Its pods keep their accelerators.
• Gives the component the ome-resources chart's default PodDisruptionBudget.

The new pods get none of LeaderWorkerSet's LWS_* variables or labels. If the runtime uses them, the pods can't find their leader, so Step 2 makes a copy of the runtime that uses OME's names.

Step 1: Find the components that run on LeaderWorkerSet¶

List the LeaderWorkerSets that OME created, in every namespace:

kubectl get leaderworkersets --all-namespaces -l ome.io/inferenceservice \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,INFERENCESERVICE:.metadata.labels.ome\.io/inferenceservice,COMPONENT:.metadata.labels.component,RUNTIME:.metadata.labels.serving-runtime'

NAMESPACE     NAME                     INFERENCESERVICE   COMPONENT   RUNTIME
deepseek-r1   lws-deepseek-r1-engine   deepseek-r1        engine      srt-deepseek-rdma

Do Step 2 once for each runtime in the list, and Steps 3 to 5 for each InferenceService. A component that also has an InferenceReplica, named <InferenceService>-<component>, has already moved: finish its move from Step 2. List the InferenceReplicas:

kubectl get inferencereplicas --all-namespaces

No resources found

Step 2: Make an OMENative copy of the runtime¶

Check whether the runtime from Step 1 uses LeaderWorkerSet's variables or labels. For a namespaced serving runtime, use servingruntime and -n with the InferenceService's namespace in this step's commands.

kubectl get clusterservingruntime srt-deepseek-rdma -o yaml \
  | grep -oE 'LWS_[A-Z_]+|leaderworkerset\.sigs\.k8s\.io/[a-z-]+' | LC_ALL=C sort -u

LWS_GROUP_SIZE
LWS_LEADER_ADDRESS
LWS_WORKER_INDEX
leaderworkerset.sigs.k8s.io/worker-index

If the command prints nothing, the runtime works on OMENative as it is: go to Step 3, and leave spec.runtime out of the change.

Don't edit the runtime in place: by default, a runtime change reaches every InferenceService that uses it, including those still on a LeaderWorkerSet. Copy it to a file instead:

kubectl get clusterservingruntime srt-deepseek-rdma -o yaml > srt-deepseek-rdma-omenative.yaml

The command prints nothing. OMENative pods use OME's names, which Pod labels and environment lists:

OME sets its variables in the pod's containers only, so an init container can't use them. Edit srt-deepseek-rdma-omenative.yaml:

1. Replace metadata with a new name:

   YAMLCopy

   metadata:
     name: srt-deepseek-rdma-omenative

2. Delete status, if the file has one.

3. Make the replacements in the table. In srt-deepseek-rdma, the variables are in the leader and worker command, and the label is in the router's --selector. The catalog's PD runtimes, srt-deepseek-rdma-pd and srt-kimi-k2-pd, have the label in --prefill-selector and --decode-selector.

4. Delete any other leaderworkerset.sigs.k8s.io key, which OMENative ignores. In place of exclusive-topology, the component's topologyKey keeps each Instance's pods in one domain, which other pods can share.

5. Delete ome.io/deploymentMode: MultiNode from engineConfig.annotations and decoderConfig.annotations, if it's there.

6. Set autoSelect: false on each entry in supportedModelFormats, as srt-deepseek-rdma already does, so that OME picks the copy only for InferenceServices that name it.

Check that nothing from LeaderWorkerSet is left:

grep -c 'LWS_\|leaderworkerset' srt-deepseek-rdma-omenative.yaml

0

Create the new runtime:

kubectl create -f srt-deepseek-rdma-omenative.yaml

clusterservingruntime.ome.io/srt-deepseek-rdma-omenative created

Step 3: Move the component to OMENative¶

Change three things in the InferenceService:

• spec.runtime.name: the runtime from Step 2.
• The ome.io/deploymentMode annotation in spec.engine.annotations: set it to OMENative, in place of the MultiNode pin.
• spec.engine.leader and spec.engine.worker: add them as empty blocks if they're missing. Empty blocks take their settings from the runtime, and make the Service select only the leaders.

Some InferenceServices need more in the same change:

Use the tab for how you manage the InferenceService:

kubectl patch inferenceservice deepseek-r1 -n deepseek-r1 --type merge \
  -p '{"spec":{"runtime":{"name":"srt-deepseek-rdma-omenative"},"engine":{"annotations":{"ome.io/deploymentMode":"OMENative"},"leader":{},"worker":{}}}}'

inferenceservice.ome.io/deepseek-r1 patched

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: deepseek-r1
  namespace: deepseek-r1
spec:
  model:
    name: deepseek-r1
  runtime:
    name: srt-deepseek-rdma-omenative
  engine:
    annotations:
      ome.io/deploymentMode: OMENative
    minReplicas: 1
    maxReplicas: 1
    leader: {}
    worker: {}

kubectl apply -f deepseek-r1.yaml

inferenceservice.ome.io/deepseek-r1 configured

Step 4: Check the new Instances¶

List the component's InferenceReplica. Its columns count Instances, not pods:

kubectl get inferencereplicas -n deepseek-r1

NAME                 COMPONENT   DESIRED   CURRENT   READY   AVAILABLE   AGE
deepseek-r1-engine   engine      1         1         1       1           3m

List the component's pods. The LeaderWorkerSet's pods, which have no ome.io/runner label, still run next to the new ones:

kubectl get pods -n deepseek-r1 -l ome.io/inferenceservice=deepseek-r1,component=engine \
  -o custom-columns='NAME:.metadata.name,RUNNER:.metadata.labels.ome\.io/runner,PHASE:.status.phase'

NAME                            RUNNER   PHASE
deepseek-r1-engine-0-leader-0   leader   Running
deepseek-r1-engine-0-worker-0   worker   Running
lws-deepseek-r1-engine-0        <none>   Running
lws-deepseek-r1-engine-0-1      <none>   Running

If the new pods stay Pending with a FailedScheduling event, the old pods hold the accelerators: do Step 5 now. Then wait for the InferenceService to be ready:

kubectl wait --for=condition=Ready inferenceservice/deepseek-r1 -n deepseek-r1 --timeout=60m

inferenceservice.ome.io/deepseek-r1 condition met

Step 5: Delete the old LeaderWorkerSet¶

Delete the LeaderWorkerSet:

kubectl delete leaderworkerset lws-deepseek-r1-engine -n deepseek-r1

leaderworkerset.leaderworkerset.x-k8s.io "lws-deepseek-r1-engine" deleted from deepseek-r1 namespace

Kubernetes deletes its pods with it. Once they've terminated, none are left:

kubectl get pods -n deepseek-r1 -l leaderworkerset.sigs.k8s.io/name=lws-deepseek-r1-engine

No resources found in deepseek-r1 namespace.

Uninstall LeaderWorkerSet¶

Once no component runs on a LeaderWorkerSet, you can uninstall LeaderWorkerSet as its docs describe. The manager checks for its CRD only when it starts, so restart the manager afterwards:

kubectl rollout restart deployment ome-controller-manager -n ome

deployment.apps/ome-controller-manager restarted

Pin a component to MultiNode¶

A component stays on its LeaderWorkerSet while its own ome.io/deploymentMode annotation says MultiNode, as How OME resolves the mode explains. Because MultiNode is deprecated, a pin is a stopgap. Pin components before you upgrade from v1.2.2, or to go back after a move. To pin the engine of deepseek-r1:

kubectl patch inferenceservice deepseek-r1 -n deepseek-r1 --type merge \
  -p '{"spec":{"engine":{"annotations":{"ome.io/deploymentMode":"MultiNode"}}}}'

inferenceservice.ome.io/deepseek-r1 patched

Pin a decoder the same way, under decoder. To pin the InferenceServices that use a runtime, add the annotation to the runtime's engineConfig.annotations or decoderConfig.annotations. This reaches only those with spec.runtime.autoSync: true, the default. If you keep InferenceServices in files, add the annotation there too.

On v1.2.2, the pin restarts the LeaderWorkerSet's pods, unless the InferenceService's metadata.annotations already has ome.io/deploymentMode: MultiNode. After the upgrade, the pods of each pinned LeaderWorkerSet restart once.

To go back after a move, pin the component and, if you changed it, set spec.runtime.name back to the old runtime, handling autoSync: false as in Step 3. OME points the Service back at the LeaderWorkerSet, which it creates again if you deleted it. If the component moved when you upgraded, the LeaderWorkerSet's pods restart once. The InferenceReplica's pods keep their accelerators until you delete it:

kubectl delete inferencereplica deepseek-r1-engine -n deepseek-r1

inferencereplica.ome.io "deepseek-r1-engine" deleted from deepseek-r1 namespace

Troubleshooting¶

The new pods never become ready¶

The usual cause is a runtime that still passes LWS_* variables. Check which ones the leader pod uses:

kubectl get pod deepseek-r1-engine-0-leader-0 -n deepseek-r1 -o yaml \
  | grep -oE 'LWS_[A-Z_]+' | LC_ALL=C sort -u

LWS_GROUP_SIZE
LWS_LEADER_ADDRESS
LWS_WORKER_INDEX

Point the component at a runtime that uses OME's names, as in Steps 2 and 3, or go back to the LeaderWorkerSet.

The Service sends requests to the workers¶

When only the runtime declares worker, the component's Service selects the workers too. The catalog's srt-deepseek-rdma gives its workers no readiness probe, so they get requests as soon as they start. Check the Service's selector:

kubectl get service deepseek-r1-engine -n deepseek-r1 -o jsonpath='{.spec.selector}{"\n"}'

{"component":"engine","ome.io/inferenceservice":"deepseek-r1","ome.io/managed-by":"OMENative"}

If the selector has no ome.io/runner, add empty leader and worker blocks to spec.engine, as in Step 3. OME then adds "ome.io/runner":"leader" to it.

The webhook rejects the change¶

kubectl prints admission webhook "inferenceservice.ome-webhook-server.validator" denied the request:, followed by this message:

InvalidDeploymentModeCombination: engine deploymentMode="OMENative" does not match decoder deploymentMode="MultiNode"; when either is OMENative, both must use the same value

The webhook reads each component's mode from the InferenceService, not from the runtime. Give the engine and the decoder the same annotation, in one patch.

The component still runs on the LeaderWorkerSet¶

After Step 3, kubectl get inferencereplicas -n deepseek-r1 shows no InferenceReplica for the component. Check the component's annotation in the InferenceService:

kubectl get inferenceservice deepseek-r1 -n deepseek-r1 \
  -o jsonpath='{.spec.engine.annotations.ome\.io/deploymentMode}{"\n"}'

MultiNode

A component moved to OMENative when you upgraded¶

The component had no ome.io/deploymentMode annotation of its own when you upgraded from v1.2.2, so it now runs on OMENative next to its old LeaderWorkerSet. Finish the move from Step 2, or pin it to MultiNode until you're ready.

Next steps¶

• Serve a multi-node model: run a new multi-node model on OMENative from the start.
• OMENative and LeaderWorkerSet: everything a component gains on OMENative.
• OMENative update strategies: choose how OME replaces the Instances' pods when the component changes.
• Let Alfred migrate Instances: let the alpha Alfred move Instances off fragmented or unhealthy nodes.