OME
ome-projects/ome
  • v1.2.2
  • 516
  • 102
↑ ↓ to move ↵ to open esc to close See all results

ome-serving chart values

Look up the ome-serving Helm chart's values for each catalog model, with their defaults, the GPU presets and the PD mode settings.

The ome-serving chart deploys models from OME's catalog, as Pre-configured models and runtimes shows. Set these values under a model's key in your values file, then run helm upgrade --install again.

Default is what the chart uses without the value. The catalog's entry for a model, in the chart's values.yaml, sets some values, such as hfModelId and runtime.gpus, and yours replace them.

Model values

Value Default Description
enabled false Deploys the model.
path None storage.path: the directory on each node that holds the weights, which the serving pods mount. It must be inside the model agent's host directory, /mnt/data/models by default.
hfModelId The catalog's A repository on Hugging Face. The storage URI becomes hf:// followed by it.
oci.namespace, oci.bucket, oci.object None An object in OCI Object Storage. The storage URI becomes oci://n/<namespace>/b/<bucket>/o/<object>.
storageUri None The storage URI itself. See Where the weights come from.
key None storage.key: the Secret with a Hugging Face token, for a gated model. The model agent reads it from the namespace ome; see Credentials.
parameters None storage.parameters, such as secretKey, the key in the Secret that holds the token.
nodeSelector, nodeAffinity None storage.nodeSelector and storage.nodeAffinity, which choose the nodes that download the model. See Choose the nodes.
vendor, capabilities The catalog's The model's vendor and modelCapabilities.
createModel true false skips the ClusterBaseModel and the BaseModel.
createRuntime true false creates no runtime, so the model needs no registry entry. OME picks another runtime that supports the model and sets autoSelect: true.
clusterScope true false skips the ClusterBaseModel, which the chart's InferenceService uses, so leave it on.
namespaceScope false true also creates a BaseModel named after the key, for InferenceServices of your own that set kind: BaseModel.
namespace The key With namespaceScope: true, the namespace of the InferenceService and the BaseModel. The chart creates it, so pick a new one for each model.
pdMode false Splits the model's serving into prefill and decode. See PD mode.

A model that the chart creates needs one of hfModelId, oci and storageUri, or the chart fails to render. With more than one, hfModelId wins, then oci. To load a catalog model from somewhere other than Hugging Face, set hfModelId: null, which removes the catalog's value, and set oci or storageUri.

With createModel: false, create a ClusterBaseModel named after the key yourself, or the webhook rejects the InferenceService. The files in config/models/ use paths under /raid/models, which work only when you install OME with modelAgent.hostPath=/raid/models, as Step 3 of Install OME explains.

Runtime values

Set these under the model's runtime key. Keep a runtime key for each model whose runtime the chart creates: without one, the chart fails to render.

Value Default Description
gpus 1 The GPUs that the engine requests, and SGLang's --tp-size. The engine's CPU and memory come from the GPU preset for this number.
image defaults.image The SGLang image, for the engine and, in PD mode, the decoder.
memFrac defaults.memFrac SGLang's --mem-frac.
extraArgs None More arguments, added at the end of SGLang's command.
routerImage defaults.routerImage The router's image.
ibDevice defaults.ibDevice In PD mode, SGLang's --disaggregation-ib-device.
rdmaProfile defaults.rdmaProfile In PD mode, the RDMA profile that OME applies to the engine and decoder pods.

Helm replaces a list instead of merging it, so a model's extraArgs in your values file replaces the catalog's. To add an argument to Phi-4-mini-instruct, list --trust-remote-code too.

Some catalog entries set cpu and memory, which the chart ignores: the engine's CPU and memory come only from its GPU preset.

Replicas

Each component's minimum and maximum replicas come from the model's values, or else from defaults.minReplicas and defaults.maxReplicas, which are both 1:

Component Values
Engine engine.minReplicas and engine.maxReplicas, or minReplicas and maxReplicas
Decoder, in PD mode decoder.minReplicas and decoder.maxReplicas
Router router.minReplicas and router.maxReplicas

Outside PD mode, the InferenceService gets a router when the model's router value is a map with at least one key, such as router: {minReplicas: 1}. The chart fails to render when engine, decoder or router isn't a map, such as router: true. The chart treats 0 as unset, so you can't set a count to 0.

Defaults

A model that doesn't set a runtime value gets it from the chart's defaults. To change it for every model, set it under defaults at the top level of your values file:

charts/ome-serving/values.yaml (excerpt)
defaults:
  image: docker.io/lmsysorg/sglang:v0.5.5.post3-cu129-amd64
  routerImage: fra.ocir.io/idqj093njucb/smg:v0.2.4.post1-dev
  memFrac: "0.9"
  minReplicas: 1
  maxReplicas: 1
  # PD (Prefill-Decode disaggregation) mode defaults
  ibDevice: mlx5_0        # InfiniBand device for RDMA
  rdmaProfile: oci-roce   # RDMA profile for network

To pull the images from your own registry, set image and routerImage to your copies. The chart has no value for image pull Secrets, and its routers run as the ServiceAccount <key>-router, not the namespace's default: see Serving pods.

GPU presets

The engine, and the decoder in PD mode, request the CPU and memory of the preset for their gpus, with limits equal to the requests:

gpus CPU Memory
1 10 30Gi
2 20 80Gi
4 20 160Gi
8 40 320Gi

A gpus value without a preset makes the chart fail to render, including the catalog's gpus: 16 for llama-4-maverick-17b-128e-instruct. To use another number of GPUs, or other resources, add or change the preset under gpuPresets in your values file, which changes it for every model that uses that number. The router's limits are always 1 CPU and 2 GiB.

PD mode

pdMode: true splits the model's serving into prefill and decode. The engine runs SGLang with --disaggregation-mode prefill, the InferenceService gets a decoder that runs it with --disaggregation-mode decode, and a router finds the engine and decoder pods by their labels. The engine and the decoder use the InfiniBand device ibDevice, so PD mode needs GPU nodes with RDMA networking that one of OME's RDMA profiles supports. Since v1.3, the profiles are oci-roce, cks-gb-sglang and cks-gb-rdma. On v1.2.2, the only one is oci-roce.

In PD mode, the engine and decoder pods:

  • use their node's network. Each one takes port 8080 on its node, so a node runs at most one engine or decoder pod in PD mode, whichever model it serves.
  • run SGLang in a privileged container, which can use all of its node's devices, with the profile oci-roce, and since v1.3 with cks-gb-sglang too.

With a profile that OME doesn't know, OME's pod webhook blocks the pods, and the component's ReplicaSet records FailedCreate events.

The chart sets no deployment mode, so the engine, decoder and router run in RawDeployment mode, where a change is a plain Deployment update. Since v1.3, OMENative can roll prefill and decode out as one group, by canary, blue-green or rolling update: see Rollout groups. Until v1.3 is released, that needs a build of main. To use it, install the model and its runtime, with pdMode: true, as in Install a model without serving it. Then create an InferenceService of your own that sets spec.deploymentMode: OMENative, as in Serve a prefill-decode model.