Size and price local models
For: cluster operators.
Use measured timings and model prices to plan local capacity.
Size a machine from its timings
The node measures each call to its model servers: how long it waited before it reached the server, its time to first token, and its prompt and decode speed (see How fast each call was). They are recorded with the call, on the node that served it, so two machines on the same model can be compared:
oz0 usage --since 168h
Speed, by model (means over the calls that measured each):
local/qwen3-coder on node-91378eb09343: first token 212 ms (140 streamed), prompt 1830 tok/s, decode 41.7 tok/s, queued 3.1 ms
local/qwen3-coder on node-d77d102ec079: first token 1.48 s (96 streamed), prompt 260 tok/s, decode 9.8 tok/s, queued 2.40 s
Read them this way:
- Queued is time before the server saw the call: mostly the wait for a free slot. A high queue with a good decode speed is a busy machine: raise its slots if its memory allows, or add a machine.
- Prompt (prefill) is how fast the server reads the prompt; long prompts, such as an agent's tools and a long conversation, make it the larger part of each step. It is bound by compute.
- Decode is how fast it writes; it is bound by memory bandwidth. A machine whose decode speed is far below another's on the same model and quantization is too small for it: pick a smaller model or quantization there.
- First token from the stream includes the prompt and the server's own scheduling.
Where the figures come from:
| Server | Prompt and decode |
|---|---|
| llama.cpp | The server's own timings (prompt and predicted milliseconds and rates), on every chat completion and on a stream's last chunk, and on the Responses API's last event |
| Ollama | The server's own timings on streamed chat completions in releases from late September 2026; otherwise measured from the stream. Ollama's own eval durations come only on its /api/chat and /api/generate, which the node does not pass on |
| vLLM, MLX, LM Studio and other OpenAI-compatible servers | Measured from the stream, unless the server sends a timings object of llama.cpp's shape: an unstreamed call has its queue time only |
The same figures are on the Prometheus histograms per model and node.
Metering
Before each call, the proxy asks the edge whether the job's and the tenant's budgets allow it, and refuses with 402 when they do not. After the call it reports what it used, and the edge records it like any call: provider local, on the node that ran it. Tokens are always recorded: the proxy turns on include_usage for streamed chat completions. A local model costs $0 unless the tenant prices it in its local record, per million tokens or per second, for example a GPU's cost:
oz0 provider model local qwen3-coder --capabilities tools --context-window 128k --per-second 0.0004
A per-second price counts from the call's upstream send to its end: the time it waited on the node, for the edge's admission and for a free slot on the server, is not charged, and neither is a call that waited too long on the fleet's model queue and reached no server.
Data rules allow local models whatever they say: the data stays on the tenant's own node.
data: local, so name the local model in model.primary, which routes the agent's jobs to the nodes that serve it. See Agents on a Mac.