Read model call timings
For: tenant administrators and cost owners.
Distinguish time waiting for capacity from time spent at the model. Older calls can lack measurements.
How fast each call was
Every model call records how long it waited and how fast the model answered, next to its tokens. The gateway measures calls through the edge, and a node measures calls to its own model servers, each on its own clock; nothing is ordered by these times.
| Figure | What it is |
|---|---|
| Queued | From the call's arrival to its upstream send: on the edge the caller's check, the budgets' reservation and a pool's moves to another member; on a node the edge's admission and the wait for a free slot on the model server. A busy machine queues; a slow one does not |
| First token | From the upstream send to the stream's first output: the first content, reasoning or tool-call delta (see below). Only streamed calls have one |
| Prompt | The prompt's processing time and its tokens per second: the server's own figures where it reports them, else the time to first token and the prompt tokens processed over it (cache reads left out) |
| Decode | The generation's time and its output tokens per second: the server's own figures where it reports them, else the time from the first output to the last, over the tokens after the first |
A figure a call did not measure is unknown, never 0: an unstreamed call has no first token, and has a prompt and decode speed only when its server reports its own. Replayed answers of the test bench and calls before this release carry none. Where the figures come from shows as the model server's own figures (llama.cpp's timings, and recent Ollama releases' on streamed chat completions) or measured from the stream.
The first output event, by format:
| Format | First output | Never counted |
|---|---|---|
| Anthropic Messages (and Bedrock's, Vertex's) | content_block_delta: text, thinking or a tool's input JSON | message_start, ping, content_block_start, a thinking block's signature_delta, message_delta |
| OpenAI chat completions | a chunk whose delta has content, reasoning_content or reasoning, refusal or tool_calls | the role-only first chunk, a null or empty content, the usage chunk, [DONE] |
| OpenAI Responses | an event whose type ends in .delta, such as response.output_text.delta, response.reasoning_text.delta or response.function_call_arguments.delta | response.created, response.in_progress, response.output_item.added, response.completed |
| Google generateContent | a response with a part that has text or a function call | an event with empty parts, the usage-only last one |
Where you see them:
oz0 job trace <job>lists each model step's last successful call under Model timings, and each call when a step took more than one; the console's trace shows them in the step's details.oz0 usageends with a Speed, by model block: the means over the period's calls that measured each figure, per model, and per node for a node's own model servers.oz0 usage --jsonhas them on each total (timedCallsandfirstTokenMs,prefillCallsandprefillTps,decodeCallsanddecodeTps,queuedCallsandqueueMs). The console's Cost page shows them under Speed by model.- Prometheus gets histograms of the first token, the decode speed and the queue.
They are figures per model and machine: none is cut by person or team. The usage export leaves them out, and so does the gateway's signed record of each call: they are measurements, not what was charged.