kubectl ome logs Unreleased: coming in v1.3

kubectl ome logs prints the logs of every pod behind an InferenceService, given only the service's name. Narrow it to one component, one OMENative Instance or one revision, and add --follow to stream new lines as they arrive. The command selects pods purely by label, so it also reads a standalone InferenceReplica's pods, given the replica's name — and finds none for a service that fronts pools through spec.replicaRefs: see Standalone and fronted replicas.

kubectl ome logs INFERENCESERVICE [flags]

The command needs list on pods and get on pods/log in the InferenceService's namespace, as Required RBAC shows.

Pods it reads¶

The command reads the pods that carry the label ome.io/inferenceservice=<name>, and never reads the InferenceService itself: the name you pass only fills the selector. On a service that declares its components, those pods are the service's engine, decoder and router. These flags narrow the selection, and you can combine them:

In prefill-decode serving, the engine runs prefill and the decoder runs decode.

Standalone and fronted replicas¶

What the ome.io/inferenceservice label carries depends on the InferenceReplica's form: the parent InferenceService's name on a replica OME projects for a service's component, but the replica's own name on a standalone replica — one you create directly, with no parent InferenceService.

So the command reads a standalone replica's pods when you pass the replica's name, where kubectl ome instance reads the InferenceService first and fails. Follow the standalone engine replica llama-pool:

kubectl ome logs llama-pool -n llama-demo -f

It streams the replica's pods. They run on OMENative and carry the same labels as a service's pods, so the narrowing flags work unchanged; a full --revision name starts with the name you pass, here llama-pool-engine-<hash>.

An InferenceService whose spec.replicaRefs fronts standalone replicas writes nothing on them and has no components of its own, so no pod anywhere carries the service's name. Given the service's name, the command always exits 1 with no pods found, even while the fronted pools serve behind its route. Read each pool with its replica's name instead. With llama-fronted fronting llama-pool:

kubectl ome logs llama-fronted -n llama-demo

error: no pods found for InferenceService "llama-fronted" in namespace "llama-demo" (selector ome.io/inferenceservice=llama-fronted)

Instances and revisions¶

--instance and --revision select only the pods that OME runs on OMENative, so they match nothing on a component that runs as a Deployment. See Deployment modes.

--instance takes an Instance's index, with --component or a full --revision. --revision takes a revision of a component's pod template: its 8-character hash, with --component, or its full name, <inferenceservice>-<component>-<hash>, which names the component. It isn't a runtime revision.

To find the indexes and hashes, list the pods with those labels as columns:

kubectl get pods -n prod -l ome.io/inferenceservice=chat -L component,ome.io/instance-index,ome.io/revision-hash

The output adds the columns COMPONENT, INSTANCE-INDEX and REVISION-HASH.

Containers¶

The command reads ome-container, OME's main container, which runs the engine or the decoder in the shipped runtimes. In a pod without it, such as a router pod, it reads the pod's only container, or fails when the pod has several. --container picks another container in every selected pod, so narrow the pods with --component too. Unlike in kubectl logs, -c here is --component, and --container has no short form.

The command reads the container's current run, or its last run while it's in CrashLoopBackOff. For the run before a restart, use kubectl logs chat-engine-0-leader-0 -n prod -c ome-container --previous.

Flags¶

Plus the standard kubeconfig flags, such as -n and --context.

The --instance default, -1, means every Instance: leave the flag out to read them all. The container fallback in the --container help is wrong: see Containers.

Output¶

Logs go to stdout. When more than one pod matches, each line starts with [<component>/<pod>], such as [engine/chat-engine-0-leader-0]. By default, each pod's whole log prints, even with --follow. --tail, --since and --limit-bytes apply to each pod: --tail 100 over three pods prints up to 300 lines.

--follow stays with the pods that matched when it started, and doesn't reopen a stream after its container restarts. Run the command again to pick up a new revision's pods.

When more pods match than --max-log-requests allows, --follow fails before it opens a stream:

error: you are attempting to follow 6 log streams, but maximum allowed concurrency is 5, use --max-log-requests to increase the limit

Raise the limit, or narrow the selection.

Examples¶

Read the logs of chat's pods once:

kubectl ome logs chat -n prod

It prints the pods' whole logs, then exits.

Read the last 10 minutes of the engine's logs, at most 100 lines from each pod:

kubectl ome logs chat -n prod -c engine --since 10m --tail 100

Read the router's logs. Router pods have one container in the shipped runtimes, so --container isn't needed:

kubectl ome logs chat -n prod -c router

Follow the engine, starting from the last 20 lines of each pod:

kubectl ome logs chat -n prod -c engine -f --tail 20

It prints new lines as they arrive, until you press Ctrl-C or all its streams end.

Follow all of chat's pods, up to 12:

kubectl ome logs chat -n prod -f --max-log-requests 12

Read the pods of the engine's Instance 0, its leader and its workers:

kubectl ome logs chat -n prod -c engine --instance 0

Read the pods of revision chat-engine-1a2b3c4d. The full name sets the component:

kubectl ome logs chat -n prod --revision chat-engine-1a2b3c4d

During a SurgeThenDrain update, a single-pod Instance runs its old and new pods at the same index. Read the pod of the decoder's single-pod Instance 0 that runs revision 5e6f7a8b:

kubectl ome logs chat -n prod -c decoder --instance 0 --revision 5e6f7a8b

Read the OME controller's logs, from every replica, with kubectl. Use your OME namespace in place of ome:

kubectl logs -n ome -l control-plane=ome-controller-manager --tail=-1

Errors¶

The other validation errors name the flag or value at fault.

Exit codes¶

Errors print to stderr as error: <message>.

Related guides¶

• Troubleshoot an InferenceService