Benchmarks

A BenchmarkJob measures how an InferenceService performs under load. OME runs genai-bench against it with the traffic shapes and concurrency levels you choose, and genai-bench saves its throughput and latency results to the storage you name. Use one to see how much load a replica takes before latency climbs, or to compare runtimes and GPU types. Run benchmarks walks through a run from start to results.

The examples use the llama-demo namespace and the llama-3-2-1b-instruct InferenceService from A minimal InferenceService.

The BenchmarkJob resource¶

A benchmark runs in iterations. Each iteration sends one traffic scenario at one concurrency level, and stops after maxTimePerIteration minutes or maxRequestsPerIteration requests, whichever comes first.

This file creates a PersistentVolumeClaim for the results, and a BenchmarkJob that runs two scenarios at three concurrency levels: six iterations of up to 10 minutes.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: benchmark-results
  namespace: llama-demo
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: ome.io/v1beta1
kind: BenchmarkJob
metadata:
  name: llama-3-2-1b-instruct-bench
  namespace: llama-demo
spec:
  endpoint:
    inferenceService:
      name: llama-3-2-1b-instruct
      namespace: llama-demo
  task: text-to-text
  trafficScenarios:
    - "D(100,100)"
    - "N(480,240)/(300,150)"
  numConcurrency:
    - 1
    - 4
    - 16
  maxTimePerIteration: 10
  maxRequestsPerIteration: 200
  outputLocation:
    storageUri: pvc://benchmark-results/llama-3-2-1b-instruct
  podOverride:
    image: ghcr.io/sgl-project/genai-bench:0.1.132

OME releases don't publish the genai-bench image that the ome-resources chart sets by default, so the example uses the image that the genai-bench project publishes. To make it the default, set the chart values ome.benchmarkJob.image to ghcr.io/sgl-project/genai-bench and ome.benchmarkJob.tag to 0.1.132. The claim uses your cluster's default StorageClass. Apply the file:

kubectl apply -f benchmark.yaml

persistentvolumeclaim/benchmark-results created
benchmarkjob.ome.io/llama-3-2-1b-instruct-bench created

The spec's fields:

additionalRequestParams and dataset have no effect: OME doesn't pass them to genai-bench. The API reference describes every field.

Endpoints¶

spec.endpoint names one target: an inferenceService or an endpoint URL. Admission rejects a BenchmarkJob that sets neither or both. You can benchmark an InferenceService whose model is stored on the nodes.

An InferenceService¶

name and namespace are required, and the InferenceService can be in another namespace. OME fills in the rest from it:

The pod runs only on nodes where the model is Ready, as Node labels and status describes. OME can't benchmark a model with a pvc:// storage URI, because no node holds a copy of it. Since v1.3, it also can't benchmark an InferenceService that names only a runtime.

An external endpoint¶

endpoint.endpoint takes a base url, an apiFormat (openai, oci-cohere or cohere) and the modelName the endpoint serves. A BenchmarkJob that targets a URL becomes Failed: genai-bench needs a tokenizer, and an API key for openai, and a BenchmarkJob has no field for either. This is a known bug.

Traffic scenarios¶

A traffic scenario sets the size of each request, in tokens or images. The forms for each task:

For example, N(480,240)/(300,150) sends around 480 input tokens and asks for around 300 output tokens, with standard deviations of 240 and 150.

Admission rejects a scenario that doesn't fit the task's forms, such as D(100). The text-to-text check is loose, so a misspelled scenario can pass admission and fail in genai-bench.

For text-to-embeddings, admission accepts only two-number scenarios, E(<n>,<n>), and rejects a BenchmarkJob that leaves trafficScenarios out. Check that your genai-bench image runs the two-number form before you rely on it. This is a known bug.

Where results go¶

outputLocation.storageUri says where genai-bench writes the results. A pvc://<claim>/<sub-path> URI writes them to that directory of a claim in the BenchmarkJob's namespace, and the claim must exist before OME creates the Job. In the example, that's the llama-3-2-1b-instruct directory of the claim benchmark-results. Run benchmarks shows how to read them. An oci://, s3://, az://, gs:// or github:// URI uploads the results to that store, with the settings in outputLocation.parameters. Benchmark output storage lists each URI format and its parameters.

resultFolderName names the folder that holds the results. Without it, genai-bench picks the name.

How a benchmark runs¶

OME creates a Job with the BenchmarkJob's name and the label benchmark: <name>, in the BenchmarkJob's namespace. For an InferenceService, OME waits until it's Ready, and creates the Job up to a minute later. The Job runs genai-bench once, and doesn't retry it. Read genai-bench's output with kubectl logs job/llama-3-2-1b-instruct-bench -n llama-demo.

The pod's defaults, and what podOverride does to each:

The ome-resources chart's ome.benchmarkJob values set the default image and resources, and a change applies to the Jobs that OME creates afterwards. The pod runs as the namespace's default ServiceAccount, so give that ServiceAccount any pull secrets it needs, as BenchmarkJob pods describes.

When you raise a request, raise its limit too. With a request above its limit, OME can't create the Job, and the BenchmarkJob stays Pending:

spec:
  podOverride:
    resources:
      requests:
        cpu: "4"
        memory: 8Gi
      limits:
        cpu: "4"
        memory: 8Gi

Editing a BenchmarkJob doesn't change its Job. To run the benchmark again, delete the Job, and OME creates a new one from the current spec. The Job and its pod stay until you delete the BenchmarkJob, and the results stay where genai-bench wrote them.

Status¶

kubectl get shows the BenchmarkJob's state:

kubectl get benchmarkjob llama-3-2-1b-instruct-bench -n llama-demo

NAME                          AGE   STATUS
llama-3-2-1b-instruct-bench   12m   Running

STATUS is status.state, which follows the Job:

The other status fields:

When a BenchmarkJob stays Pending, the controller logs say why:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1 | grep llama-3-2-1b-instruct-bench

The BenchmarkJob stays Pending lists the common messages and their fixes. A request above its limit, or an output URI from URIs OME can't write to, also keeps a BenchmarkJob Pending.

Next steps¶

• Run benchmarks: run a BenchmarkJob, and read its results.
• Benchmark output storage: the URI formats and parameters for each backend.