OME
ome-projects/ome
  • v1.2.2
  • 516
  • 102
↑ ↓ to move ↵ to open esc to close See all results

Pace rollouts with minReadySeconds Unreleased: coming in v1.3

Make a new OMENative pod stay Ready for a warm-up window before it counts as Available, so a rollout waits before draining the old pods.

A model server can pass its readiness probe and still fail while it warms up under its first requests. lifecycle.minReadySeconds gives each new pod a warm-up window. In a rollout with the default strategy, OME keeps the old pods serving until the new ones have stayed Ready for the whole window. It works like a Deployment's minReadySeconds, for components that run on OMENative.

You can set it on an InferenceService, or as a default in a runtime or for the whole cluster.

Ready versus available

Each replica of an OMENative component is an Instance: one pod, or a leader and its workers. A new pod is Ready, and receives requests, once its containers pass their readiness checks and OME puts it into rotation. It's Available once it has stayed Ready for minReadySeconds. The window delays what OME does next:

When What waits for the window
A SurgeThenDrain rollout, the default Draining the old pods, which keep serving until then.
A RecreatePod, InPlaceIfPossible or InPlaceOnly rollout The updated Instance's phase turning Ready on the new revision.
First deploy and scale-up The new Instance's phase turning Ready.
An Instance migration Draining the source Instance, which keeps serving until then. The window counts toward the migration's deadline.

The InferenceService's Ready condition doesn't wait for the window. An Instance that's being replaced holds its place in the maxSurge or maxUnavailable budget for the whole window. So a rollout takes at least one window longer for each batch of Instances that move together. If a new pod stops being Ready during its window, the window starts over. Readiness and availability defines the states.

Step 1: Set a window on an InferenceService

Create an InferenceService that runs in OMENative mode, with a 900-second window on its engine:

isvc.yaml
apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: llama-demo
spec:
  deploymentMode: OMENative
  model:
    name: llama-3-2-1b-instruct
  runtime:
    name: srt-llama-3-2-1b-instruct
  engine:
    minReplicas: 1
    maxReplicas: 1
    lifecycle:
      minReadySeconds: 900

spec.deploymentMode: OMENative puts every component of the service on OMENative. In RawDeployment, MultiNode (deprecated) or any other mode, OME ignores minReadySeconds, and the webhook admits it without a warning. How OME resolves the mode lists what else sets a component's mode.

Make the window longer than your readiness probe takes to catch a failure: about failureThreshold times the longer of periodSeconds and timeoutSeconds. Otherwise a pod can become Available before a failing probe marks it not Ready. The engine of srt-llama-3-2-1b-instruct probes every 60 seconds, waits up to 200 seconds for an answer and allows 3 failures. It catches a server that returns errors in about 3 minutes and one that hangs in about 10, so 900 seconds covers both.

Apply the file:

Shell
kubectl apply -f isvc.yaml
Output
inferenceservice.ome.io/llama-chat created

The first start pulls the SGLang image and loads the model, which can take several minutes.

Step 2: Check the window in effect

OME writes an InferenceReplica for each OMENative component, named <service>-<component>, here llama-chat-engine. Its spec.minReadySeconds is the window in effect, wherever it's set. Its status counts the Instances whose pods are all Ready, and those that are Available:

Shell
kubectl get inferencereplica llama-chat-engine -n llama-demo \
  -o custom-columns=NAME:.metadata.name,MIN-READY:.spec.minReadySeconds,READY:.status.readyReplicas,AVAILABLE:.status.availableReplicas

During the window, the output looks like this:

Output
NAME                MIN-READY   READY   AVAILABLE
llama-chat-engine   900         1       <none>

<none> means 0. Once the pod has been Ready for 900 seconds, AVAILABLE shows 1. The InferenceService reports the same counts in status.components.engine.lifecycle. Its spec shows only the value you set there.

Set a default for many services

OME takes each component's window from the first of these that sets it, and uses 0 when none does:

Set in Field
The InferenceService spec.engine.lifecycle.minReadySeconds, or the same under decoder or router. It wins even when it's 0.
The service's ServingRuntime or ClusterServingRuntime spec.engineConfig.lifecycle.minReadySeconds, or the same under decoderConfig or routerConfig.
The cluster ome.controller.minReadySeconds in the ome-resources chart, unset by default.

The value is a whole number of seconds, 0 or more.

In a runtime

Set a 1200-second default on the engine of srt-llama-3-2-1b-instruct:

Shell
kubectl patch clusterservingruntime srt-llama-3-2-1b-instruct --type merge \
  -p '{"spec":{"engineConfig":{"lifecycle":{"minReadySeconds":1200}}}}'
Output
clusterservingruntime.ome.io/srt-llama-3-2-1b-instruct patched

llama-chat sets its own value, so it keeps 900. Remove it, and the runtime's default takes over:

Shell
kubectl patch inferenceservice llama-chat -n llama-demo --type merge \
  -p '{"spec":{"engine":{"lifecycle":{"minReadySeconds":null}}}}'
Output
inferenceservice.ome.io/llama-chat patched
Shell
kubectl get inferencereplica llama-chat-engine -n llama-demo -o jsonpath='{.spec.minReadySeconds}'
Output
1200

A service can pin its runtime with spec.runtime.autoSync: false. When you change the runtime, OME sets that service's RuntimeDrifted condition to True and leaves its serving workload as it is until you move the pin. See Runtime revisions and pinning.

For the whole cluster

Add the default to the values file that you installed ome-resources with:

values.yaml
ome:
  controller:
    minReadySeconds: 60

Keep your other values in the file: helm upgrade resets every value that isn't in it to the chart's default. Upgrade with the chart version that you already run, so that OME stays in step with the ome-crd chart. helm list -n ome shows it in its CHART column:

Shell
helm upgrade --install ome oci://ghcr.io/moirai-internal/charts/ome-resources \
  --namespace ome --version 1.3.0 -f values.yaml

Until v1.3 is released, upgrade from a checkout of main instead, as in Install from source.

Helm reports that the release has been upgraded. The upgrade restarts the controller, which then applies the new default. Check the deploy entry of the inferenceservice-config ConfigMap:

Shell
kubectl get configmap inferenceservice-config -n ome -o jsonpath='{.data.deploy}' | grep minReadySeconds
Output
  "minReadySeconds": 60,

If you manage the ConfigMap yourself, add "minReadySeconds": 60 to the JSON in its deploy entry. A later helm upgrade can overwrite the edit. Change ConfigMap settings explains when an edit takes effect, and how to apply it at once.

Troubleshooting

The controller crash-loops or services stop updating

Nothing checks the cluster default when you set it. With a negative value, such as -30, new controller pods exit with invalid deploy config, minReadySeconds must be >= 0, got -30. A running controller logs the same error for every InferenceService and stops creating or updating serving workloads. Read the logs of every replica, and look for that error:

Shell
kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

Set the value to 0 or more in your values file or the ConfigMap, then upgrade again or restart the controller.

The value doesn't seem to apply

Check spec.minReadySeconds on the component's InferenceReplica, as in Step 2, then find your case:

What you see Cause and fix
kubectl reports that the InferenceReplica isn't found The component doesn't run on OMENative. Set spec.deploymentMode: OMENative, and remove any ome.io/deploymentMode annotation that names another mode, on the component or in the runtime.
The value isn't the runtime's The InferenceService sets its own value, even 0. Or it pins its runtime with spec.runtime.autoSync: false, and gets the change once you move the pin.
Pods didn't restart Expected: the window isn't part of the pod template. OME uses a new value from then on, in rollouts in progress too.

Clean up

Delete the InferenceService:

Shell
kubectl delete inferenceservice llama-chat -n llama-demo
Output
inferenceservice.ome.io "llama-chat" deleted from llama-demo namespace

If you set the runtime default, remove it:

Shell
kubectl patch clusterservingruntime srt-llama-3-2-1b-instruct --type merge \
  -p '{"spec":{"engineConfig":{"lifecycle":{"minReadySeconds":null}}}}'
Output
clusterservingruntime.ome.io/srt-llama-3-2-1b-instruct patched

If you set the cluster default, remove minReadySeconds from your values file, and run the helm upgrade command again with the same version.

Next steps

  • OMENative update strategies: how SurgeThenDrain and the other strategies replace an Instance's pods, and how partition, maxSurge and maxUnavailable pace a rollout.
  • Rollout groups: roll the engine and the decoder out as one, by canary, blue-green or rolling update.
  • Pause and resume a rollout: hold a rollout in progress with the alpha kubectl ome rollout pause action, and continue it with resume.
  • LifecycleSpec in the API reference: the other lifecycle fields of a component.