Benchmark output storage

A BenchmarkJob writes its results to the location in spec.outputLocation, a required StorageSpec. OME reads two of its fields, and turns both into arguments for genai-bench, the benchmark tool that runs in the BenchmarkJob's Job:

• storageUri, whose scheme picks the backend;
• parameters, a map of settings and credentials for that backend.

OME ignores the other StorageSpec fields. None of them, key included, points outputLocation at a Secret.

For a claim, OME mounts it in the pod, and genai-bench writes to it directly. For the other five backends, OME passes --upload-results and the location, and genai-bench uploads the results there.

How OME checks a URI and its parameters:

• OME's BenchmarkJob webhook checks the URI's form when you create or update a BenchmarkJob, and rejects an unknown scheme or a malformed URI. Its message starts with invalid storage: error parsing storage URI:, followed by the reason. For an unknown scheme, the reason is unknown storage type for URI: <uri>. Each section below lists the reasons for its backend.
• Nothing checks that the bucket, container, repository or credentials work. genai-bench finds out when it writes.
• OME matches parameter keys exactly, case included. It ignores keys it doesn't know, and the keys of other backends, without a warning. Admission doesn't check them either.
• Admission also accepts well-formed hf://, vendor:// and local:// URIs, which name model sources elsewhere in OME, but OME can't write results to them. See URIs OME can't write to.

Credentials appear on the pod command line¶

OME has no Secret reference for storage credentials. It copies the value of each parameter it knows, as it is, into the arguments of the genai-bench container. These parameters hold credentials:

A value you set is in three objects in the BenchmarkJob's namespace:

• the BenchmarkJob, in spec.outputLocation.parameters;
• its Job, in the container args of the pod template;
• the Job's pod, in the container args.

OME sets no ttlSecondsAfterFinished on the Job, so the Job and its pod keep the values after the benchmark ends, until you delete the BenchmarkJob.

Anyone who can read one of these objects can read the values:

• The built-in view ClusterRole reads pods and Jobs, although it leaves Secrets out.
• Since v1.3, the ome-resources chart adds read access to every ome.io resource to view, through its ome-supplemental-viewer ClusterRole, so view reads BenchmarkJobs too. On v1.2.2, view doesn't read BenchmarkJobs, but it still reads the Job and the pod.

To see what the pod spec holds, print the container arguments of the Job:

kubectl get job llama-3-2-1b-instruct-bench -n llama-demo \
  -o jsonpath='{.spec.template.spec.containers[0].args}'

It prints the arguments as a JSON array: the benchmark settings, then the storage arguments that the sections below list, with each parameter value after its flag.

To keep a credential out of the pod spec:

• Where a parameter takes a path, config_file for OCI Object Storage or gcp_credentials_path for Google Cloud Storage, mount a Secret as a file with the volumes and volumeMounts of podOverride, and pass the file's path. The Google Cloud Storage example does this. OME doesn't check that the file exists.
• Or leave the credential parameters out. OME then passes no credentials, and whether the upload works depends on what genai-bench finds in the pod. The pod runs as the default ServiceAccount of the BenchmarkJob's namespace, and podOverride has no field for another one: see BenchmarkJob pods.

OCI Object Storage¶

oci://n/<namespace>/b/<bucket>/o/<prefix>

<namespace> is your tenancy's Object Storage namespace, not a Kubernetes namespace. The prefix can span several path segments. OME passes:

• --upload-results;
• --namespace <namespace>, --storage-bucket <bucket> and --storage-prefix <prefix>, even when the prefix is empty.

Unlike for the other backends that upload, OME passes no --storage-provider.

Admission rejects a URI that has fewer than six path segments after oci://, or doesn't have n, b and o as its first, third and fifth, with invalid OCI storage URI format. Expected: oci://n/{namespace}/b/{bucket}/o/{object_path}.

This outputLocation uploads to the bucket benchmark-results:

spec:
  outputLocation:
    storageUri: oci://n/examplens/b/benchmark-results/o/llama-3-2-1b-instruct
    parameters:
      auth: instance_principal
      region: eu-frankfurt-1

OME appends these arguments to the args of the genai-bench container:

--upload-results
--namespace examplens
--storage-bucket benchmark-results
--storage-prefix llama-3-2-1b-instruct
--auth instance_principal
--region eu-frankfurt-1

AWS S3¶

s3://<bucket>/<prefix>
s3://<bucket>@<region>/<prefix>

The prefix is optional, as in s3://benchmark-results. OME passes:

• --upload-results and --storage-provider aws;
• --storage-bucket <bucket>, and --storage-prefix <prefix> when the URI has a prefix;
• --storage-aws-region, from the aws_region parameter, or else from @<region>. With neither, OME passes no region.

OME reads the first @ in the URI as the start of the region, wherever it is, so don't put @ in the prefix: s3://benchmark-results/runs@2026/llama gives the bucket benchmark-results/runs, the region 2026 and the prefix llama.

Admission rejects s3:// with invalid S3 storage URI format: missing bucket name, and an empty bucket, as in s3:///llama-3-2-1b-instruct, with invalid S3 storage URI format: bucket name cannot be empty.

This outputLocation uploads to the bucket benchmark-results in us-west-2:

spec:
  outputLocation:
    storageUri: s3://benchmark-results/llama-3-2-1b-instruct
    parameters:
      aws_region: us-west-2

OME appends these arguments to the args of the genai-bench container:

--upload-results
--storage-provider aws
--storage-bucket benchmark-results
--storage-prefix llama-3-2-1b-instruct
--storage-aws-region us-west-2

Azure Blob Storage¶

az://<account>/<container>/<path>
az://<account>.blob.core.windows.net/<container>/<path>

The two forms name the same location, and the path is optional. OME passes:

• --upload-results and --storage-provider azure;
• --storage-bucket <container>, and --storage-prefix <path> when the URI has a path;
• always --storage-azure-account-name, from the azure_account_name parameter, or else from the URI.

Admission rejects:

• az://, with invalid Azure storage URI format: missing account name;
• a URI without a container, as in az://benchmarkstore, with invalid Azure storage URI format: missing container name;
• an empty account or container, as in az://benchmarkstore//llama-3-2-1b-instruct, with invalid Azure storage URI format: account name and container name are required.

This outputLocation uploads to the container benchmark-results of the storage account benchmarkstore:

spec:
  outputLocation:
    storageUri: az://benchmarkstore/benchmark-results/llama-3-2-1b-instruct

OME appends these arguments to the args of the genai-bench container:

--upload-results
--storage-provider azure
--storage-bucket benchmark-results
--storage-prefix llama-3-2-1b-instruct
--storage-azure-account-name benchmarkstore

Google Cloud Storage¶

gs://<bucket>/<path>

The path is optional. OME passes --upload-results, --storage-provider gcp and --storage-bucket <bucket>, and --storage-prefix <path> when the URI has a path.

Admission rejects gs:// with invalid GCS storage URI format: missing bucket name, and an empty bucket, as in gs:///llama-3-2-1b-instruct, with invalid GCS storage URI format: bucket name cannot be empty.

This BenchmarkJob uploads to the bucket benchmark-results. It mounts a service account key from the key.json key of the Secret gcs-benchmark-writer, and passes only the key's path:

apiVersion: ome.io/v1beta1
kind: BenchmarkJob
metadata:
  name: llama-3-2-1b-instruct-bench
  namespace: llama-demo
spec:
  endpoint:
    inferenceService:
      name: llama-3-2-1b-instruct
      namespace: llama-demo
  task: text-to-text
  maxTimePerIteration: 10
  maxRequestsPerIteration: 200
  outputLocation:
    storageUri: gs://benchmark-results/llama-3-2-1b-instruct
    parameters:
      gcp_project_id: llama-demo-project
      gcp_credentials_path: /var/run/secrets/gcs/key.json
  podOverride:
    volumes:
      - name: gcs-credentials
        secret:
          secretName: gcs-benchmark-writer
    volumeMounts:
      - name: gcs-credentials
        mountPath: /var/run/secrets/gcs
        readOnly: true

OME appends these arguments to the args of the genai-bench container:

--upload-results
--storage-provider gcp
--storage-bucket benchmark-results
--storage-prefix llama-3-2-1b-instruct
--storage-gcp-project-id llama-demo-project
--storage-gcp-credentials-path /var/run/secrets/gcs/key.json

GitHub Releases¶

github://<owner>/<repository>
github://<owner>/<repository>@<tag>

The tag names the release. OME passes:

• --upload-results and --storage-provider github;
• --github-owner <owner> and --github-repo <repository>;
• --github-tag <tag>, unless the tag is latest. A URI without a tag counts as latest, so for both, OME passes no tag and leaves the choice of release to genai-bench.

Admission rejects:

• github://, with invalid GitHub storage URI format: missing owner/repository;
• a URI without a slash, as in github://acme, with invalid GitHub storage URI format: expected owner/repository;
• an empty owner or repository, as in github://acme/, with invalid GitHub storage URI format: owner and repository are required.

This outputLocation uploads to the release llama-3-2-1b-instruct of the repository acme/benchmarks:

spec:
  outputLocation:
    storageUri: github://acme/benchmarks@llama-3-2-1b-instruct

OME appends these arguments to the args of the genai-bench container:

--upload-results
--storage-provider github
--github-owner acme
--github-repo benchmarks
--github-tag llama-3-2-1b-instruct

With github_token set, OME also appends --github-token and the token.

Persistent volume claims¶

pvc://<claim>/<sub-path>

For a claim, OME:

• looks up the claim in the BenchmarkJob's namespace before it creates the Job;
• adds a volume named benchmark-output-storage for the claim, and mounts the claim's <sub-path> directory at /<sub-path> in the genai-bench container;
• passes --experiment-base-dir /<sub-path> and no --upload-results, so genai-bench writes the results to the claim, and nothing is uploaded;
• ignores parameters.

The parser also accepts pvc://<namespace>:<claim>/<sub-path>, but OME ignores the namespace, since a pod can mount only claims in its own namespace.

The kubelet creates the sub-path directory in the claim when it's missing. The mount hides whatever the image has at /<sub-path>, so pick a sub-path that doesn't name one of the image's directories, such as usr. A podOverride volume named benchmark-output-storage, or a podOverride mount at /<sub-path>, overrides OME's.

This outputLocation, from the example on the Benchmarks page, writes to the llama-3-2-1b-instruct directory of the claim benchmark-results:

spec:
  outputLocation:
    storageUri: pvc://benchmark-results/llama-3-2-1b-instruct

OME mounts that directory at /llama-3-2-1b-instruct, and appends this argument to the args of the genai-bench container:

--experiment-base-dir /llama-3-2-1b-instruct

These URIs are rejected:

Since v1.3, admission rejects a sub-path with a .. segment. On v1.2.2, admission accepts it, the API server rejects the Job with must not contain '..', and the BenchmarkJob stays Pending, as for the sub-path that starts with / in URIs OME can't write to.

URIs OME can't write to¶

Admission accepts these URIs, but OME can't create a Job for them:

In each case:

• OME creates no Job, and the BenchmarkJob stays Pending. The status doesn't say why, and OME records no event.
• Each attempt logs a Reconciler error line with the error, and OME tries again after a growing delay. Status on the Benchmarks page shows how to read the logs of every controller replica.
• For an InferenceService endpoint, OME builds the Job only once the InferenceService is Ready, so the error appears after that.

While no Job exists, you can fix outputLocation in place, or create the missing claim, and OME's next attempt uses it. An edit to the BenchmarkJob starts an attempt right away. Once the Job exists, OME doesn't update it. To apply an edited outputLocation, delete the Job, as on the Benchmarks page: OME creates a new one from the current spec.

A URI that OME accepts can still point at storage that genai-bench can't write to, such as a bucket that doesn't exist, credentials without access, or an endpoint the pod can't reach. OME checks none of these. genai-bench's errors are in the pod's logs, which kubectl logs job/llama-3-2-1b-instruct-bench -n llama-demo prints. If genai-bench exits with an error, the Job fails without a retry, and the BenchmarkJob's state becomes Failed.

Related pages¶

• Where results go: how a BenchmarkJob uses outputLocation, and the resultFolderName that names the results folder.
• How a benchmark runs: the Job and pod that OME creates, and what podOverride changes.
• Run benchmarks: run a BenchmarkJob against an InferenceService, and collect its results.
• Serve models from node-local storage: local:// URIs, which name model sources, not benchmark output.
• OME API: the StorageSpec type.