---
title: "Size and price local models"
description: "Use measured timings and model prices to plan local capacity."
canonical_url: "https://docs.oz0.ai/fleet/local-models/sizing-and-cost"
---
# Size and price local models

> Use measured timings and model prices to plan local capacity.

For: cluster operators.

Use measured timings and model prices to plan local capacity.

## Size a machine from its timings

The node measures each call to its model servers: how long it waited before it reached the server, its time to first token, and its prompt and decode speed (see [How fast each call was](https://docs.oz0.ai/spend/cost-and-usage/timings#how-fast-each-call-was)). They are recorded with the call, on the node that served it, so two machines on the same model can be compared:

```bash [Terminal]
oz0 usage --since 168h
```

```text
Speed, by model (means over the calls that measured each):
  local/qwen3-coder on node-91378eb09343: first token 212 ms (140 streamed), prompt 1830 tok/s, decode 41.7 tok/s, queued 3.1 ms
  local/qwen3-coder on node-d77d102ec079: first token 1.48 s (96 streamed), prompt 260 tok/s, decode 9.8 tok/s, queued 2.40 s
```

Read them this way:

- Queued is time before the server saw the call: mostly the wait for a free slot. A high queue with a good decode speed is a busy machine: raise its slots if its memory allows, or add a machine.
- Prompt (prefill) is how fast the server reads the prompt; long prompts, such as an agent's tools and a long conversation, make it the larger part of each step. It is bound by compute.
- Decode is how fast it writes; it is bound by memory bandwidth. A machine whose decode speed is far below another's on the same model and quantization is too small for it: pick a smaller model or quantization there.
- First token from the stream includes the prompt and the server's own scheduling.

Where the figures come from:

<table>
<thead>
  <tr>
    <th>
      Server
    </th>
    
    <th>
      Prompt and decode
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      llama.cpp
    </td>
    
    <td>
      The server's own <code>
        timings
      </code>
      
       (prompt and predicted milliseconds and rates), on every chat completion and on a stream's last chunk, and on the Responses API's last event
    </td>
  </tr>
  
  <tr>
    <td>
      Ollama
    </td>
    
    <td>
      The server's own <code>
        timings
      </code>
      
       on streamed chat completions in releases from late September 2026; otherwise measured from the stream. Ollama's own eval durations come only on its <code>
        /api/chat
      </code>
      
       and <code>
        /api/generate
      </code>
      
      , which the node does not pass on
    </td>
  </tr>
  
  <tr>
    <td>
      vLLM, MLX, LM Studio and other OpenAI-compatible servers
    </td>
    
    <td>
      Measured from the stream, unless the server sends a <code>
        timings
      </code>
      
       object of llama.cpp's shape: an unstreamed call has its queue time only
    </td>
  </tr>
</tbody>
</table>

The same figures are on the Prometheus [histograms](https://docs.oz0.ai/fleet/observability#metrics) per model and node.

## Metering

Before each call, the proxy asks the edge whether the job's and the tenant's budgets allow it, and refuses with `402` when they do not. After the call it reports what it used, and the edge records it like any call: provider `local`, on the node that ran it. Tokens are always recorded: the proxy turns on `include_usage` for streamed chat completions. A local model costs $0 unless the tenant prices it in its `local` record, per million tokens or per second, for example a GPU's cost:

```bash [Terminal]
oz0 provider model local qwen3-coder --capabilities tools --context-window 128k --per-second 0.0004
```

A per-second price counts from the call's upstream send to its end: the time it waited on the node, for the edge's admission and for a free slot on the server, is not charged, and neither is a call that waited too long on the fleet's model queue and reached no server.

Data rules allow local models whatever they say: the data stays on the tenant's own node.

<note>

A harness on a local model takes the server's format, as above: Codex needs Responses and the Claude Agent SDK Messages. Harness agents cannot use `data: local`, so name the local model in `model.primary`, which routes the agent's jobs to the nodes that serve it. See [Agents on a Mac](https://docs.oz0.ai/fleet/local-models-on-a-mac#which-harness-on-which-server).

</note>


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.