Run benchmarks

A BenchmarkJob measures the throughput and latency of an InferenceService under load. OME runs genai-bench in a Kubernetes Job with the traffic scenarios and concurrency levels you choose, and genai-bench saves the results to storage. Here you benchmark the qwen3-0-6b InferenceService from Serve your first model, and copy the results from a PersistentVolumeClaim.

Step 1: Create a benchmark¶

Save this file as qwen3-0-6b-bench.yaml. It creates a claim for the results, and a BenchmarkJob that runs two traffic scenarios at two concurrency levels, four iterations in all:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: benchmark-results
  namespace: qwen3-0-6b
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: ome.io/v1beta1
kind: BenchmarkJob
metadata:
  name: qwen3-0-6b-bench
  namespace: qwen3-0-6b
spec:
  endpoint:
    inferenceService:
      name: qwen3-0-6b
      namespace: qwen3-0-6b
  task: text-to-text
  trafficScenarios:
    - "D(100,100)"
    - "D(2000,200)"
  numConcurrency:
    - 1
    - 8
  maxTimePerIteration: 5
  maxRequestsPerIteration: 100
  outputLocation:
    storageUri: pvc://benchmark-results/qwen3-0-6b
  resultFolderName: baseline
  podOverride:
    image: ghcr.io/sgl-project/genai-bench:0.1.132

OME starts the benchmark once the InferenceService is Ready. The benchmark pod must reach the InferenceService's URL. With ingress creation off, the default, that's the Service's address inside the cluster. Each request also names the model. OME reads the name from the engine pods: here it's Qwen/Qwen3-0.6B, from the runtime's --served-model-name, the name you used in Step 7 of Serve your first model.

endpoint can name a URL instead of an InferenceService, but such a benchmark fails, as An external endpoint explains. The BenchmarkJob resource lists every field.

Apply the file:

kubectl apply -f qwen3-0-6b-bench.yaml

persistentvolumeclaim/benchmark-results created
benchmarkjob.ome.io/qwen3-0-6b-bench created

Step 2: Watch it run¶

Check the BenchmarkJob's state:

kubectl get benchmarkjob qwen3-0-6b-bench -n qwen3-0-6b

NAME               AGE   STATUS
qwen3-0-6b-bench   3m    Running

STATUS is Pending until OME creates the Job, then Running, and then Completed or Failed. If it stays Pending, see The BenchmarkJob stays Pending.

The Job that runs genai-bench has the BenchmarkJob's name. Follow its output:

kubectl logs -f job/qwen3-0-6b-bench -n qwen3-0-6b

It prints genai-bench's output until genai-bench exits.

Or wait for the state Completed. The four iterations take up to about 20 minutes, plus the time to pull the image and start genai-bench:

kubectl wait --for=jsonpath='{.status.state}'=Completed benchmarkjob/qwen3-0-6b-bench -n qwen3-0-6b --timeout=40m

benchmarkjob.ome.io/qwen3-0-6b-bench condition met

If the state becomes Failed, the command waits until the timeout. See The BenchmarkJob fails.

Step 3: Read the results¶

genai-bench wrote the results to qwen3-0-6b/baseline on the claim. The benchmark pod has stopped, so read them from a pod that mounts the claim. Save this file as benchmark-results-reader.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: benchmark-results-reader
  namespace: qwen3-0-6b
spec:
  containers:
    - name: reader
      image: busybox:1.37
      command: ["sleep", "3600"]
      volumeMounts:
        - name: results
          mountPath: /results
          readOnly: true
  volumes:
    - name: results
      persistentVolumeClaim:
        claimName: benchmark-results
        readOnly: true

Create the pod, which mounts the claim read-only at /results, and wait for it to be ready:

kubectl apply -f benchmark-results-reader.yaml
kubectl wait --for=condition=Ready pod/benchmark-results-reader -n qwen3-0-6b --timeout=5m

pod/benchmark-results-reader created
pod/benchmark-results-reader condition met

List the results:

kubectl exec -n qwen3-0-6b benchmark-results-reader -- ls /results/qwen3-0-6b/baseline

The files, and what they hold, depend on the genai-bench version: see genai-bench's documentation.

Copy the folder to your machine:

kubectl cp qwen3-0-6b/benchmark-results-reader:/results/qwen3-0-6b/baseline ./baseline

kubectl copies the folder to baseline in your working directory.

Run the benchmark again¶

OME runs a BenchmarkJob's Job once, and doesn't change it when you edit the BenchmarkJob. To run again, edit the BenchmarkJob, with a new resultFolderName so the results go to a new folder. Then delete the Job:

kubectl delete job qwen3-0-6b-bench -n qwen3-0-6b

job.batch "qwen3-0-6b-bench" deleted from qwen3-0-6b namespace

OME creates a new Job from the current spec. The BenchmarkJob is Pending until it does.

Choose traffic scenarios¶

A BenchmarkJob runs each traffic scenario at each concurrency level. For text-to-text, a scenario sets how many tokens each request sends and asks for:

Traffic scenarios lists the forms for every task. Admission checks text-to-text scenarios loosely, so a typo can pass kubectl apply and fail later in genai-bench.

numConcurrency sets how many requests genai-bench keeps in flight. Run a few levels, from 1 to beyond the load you expect, to see how throughput and latency change as the load grows.

If you leave out trafficScenarios or numConcurrency, genai-bench uses its own defaults for that setting.

For a text-to-embeddings model, set trafficScenarios in the form E(<n>,<n>), the only form admission accepts. Read Traffic scenarios first.

Troubleshooting¶

Admission rejects the BenchmarkJob¶

For the scenario D(100), kubectl apply prints this line:

Error from server (Forbidden): error when creating "qwen3-0-6b-bench.yaml": admission webhook "benchmarkjob.ome-webhook-server.validator" denied the request: invalid traffic scenarios: failed to validate scenario 'D(100)': invalid scenario format for task 'text-to-text': D(100)

The webhook's reason, after denied the request:, starts with the part of the spec that's wrong:

The BenchmarkJob stays Pending¶

The reason is in the controller's logs, not in the BenchmarkJob's status. Search the logs of every controller replica for the BenchmarkJob's name. The command assumes that OME runs in the ome namespace:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep qwen3-0-6b-bench

Look for these messages:

Status lists the other causes. OME retries with a growing delay. After you fix the cause, annotate the BenchmarkJob to make it try again right away:

kubectl annotate benchmarkjob qwen3-0-6b-bench -n qwen3-0-6b retry=1 --overwrite

benchmarkjob.ome.io/qwen3-0-6b-bench annotated

The benchmark pod doesn't start¶

The BenchmarkJob is Running, but its pod stays Pending or can't pull its image. The events at the end of the pod's description say why:

kubectl describe pod -n qwen3-0-6b -l benchmark=qwen3-0-6b-bench

• FailedScheduling: no node where the model is Ready has room for the pod, or the node has a taint that the pod doesn't tolerate. Free room on such a node, or set podOverride.resources or podOverride.tolerations. Your tolerations replace the default nvidia.com/gpu one, so list every taint that the pod must tolerate.
• ErrImagePull or ImagePullBackOff: check that podOverride.image is set as in Step 1. If the registry refuses access, give the default ServiceAccount in qwen3-0-6b a pull secret, as BenchmarkJob pods describes.

The pod picks up podOverride changes and pull secrets only when it's created, so after those fixes, run the benchmark again.

The BenchmarkJob fails¶

genai-bench exited with an error, and the Job doesn't retry it. The end of the pod's log holds the error:

kubectl logs job/qwen3-0-6b-bench -n qwen3-0-6b

Fix the cause in the BenchmarkJob or at the endpoint, then run the benchmark again.

Clean up¶

Delete the BenchmarkJob, which also deletes its Job and pod, then the reader pod:

kubectl delete benchmarkjob qwen3-0-6b-bench -n qwen3-0-6b
kubectl delete pod benchmark-results-reader -n qwen3-0-6b

benchmarkjob.ome.io "qwen3-0-6b-bench" deleted from qwen3-0-6b namespace
pod "benchmark-results-reader" deleted from qwen3-0-6b namespace

The results stay on the claim. Delete the claim when you no longer need them. With a StorageClass whose reclaim policy is Delete, that deletes the volume, and the results with it:

kubectl delete pvc benchmark-results -n qwen3-0-6b

persistentvolumeclaim "benchmark-results" deleted from qwen3-0-6b namespace

To delete the InferenceService, the runtime and the model too, follow Clean up in Serve your first model.

Next steps¶

• Benchmarks: every field of a BenchmarkJob, the rules for endpoints and traffic scenarios, and the status.
• Benchmark output storage: write the results to OCI Object Storage, Amazon S3, Azure Blob Storage, Google Cloud Storage or GitHub Releases.