---
title: "Configure local models"
description: "Configure a node-local model server and name its model in an agent."
canonical_url: "https://docs.oz0.ai/fleet/local-models"
---
# Configure local models

> Configure a node-local model server and name its model in an agent.

For: cluster operators.

A model server on a node is a capability of that node. A laptop with Ollama, a Mac with MLX or LM Studio ([Agents on a Mac](https://docs.oz0.ai/fleet/local-models-on-a-mac)), or a GPU box with vLLM, becomes a node that agents on local models are routed to, under the same budgets and in the same cost log as cloud models.

<table>
<thead>
  <tr>
    <th>
      Where the model runs
    </th>
    
    <th>
      How
    </th>
    
    <th>
      Status
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      A shared server next to the edge, such as a vLLM cluster
    </td>
    
    <td>
      An ordinary <a href="https://docs.oz0.ai/spend/models-and-keys#providers-are-records">
        provider record
      </a>
      
      : <code>
        provider set vllm --preset vllm --base-url ... --internal
      </code>
    </td>
    
    <td>
      Works today
    </td>
  </tr>
  
  <tr>
    <td>
      A server on the node itself
    </td>
    
    <td>
      <code>
        local/&lt;model>
      </code>
      
      : this page
    </td>
    
    <td>
      Works today
    </td>
  </tr>
  
  <tr>
    <td>
      A GPU node that serves the whole fleet
    </td>
    
    <td>
      <code>
        model.placement: fleet
      </code>
      
      : a queue per model that only its GPU nodes poll (<a href="https://docs.oz0.ai/fleet/local-models/fleet#share-a-gpu-with-the-fleet">
        in this guide
      </a>
      
      )
    </td>
    
    <td>
      Works today
    </td>
  </tr>
</tbody>
</table>

## Choose a task

<table>
<thead>
  <tr>
    <th>
      Task
    </th>
    
    <th>
      Guide
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Share a local model with the fleet
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/local-models/fleet">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Review local-model safety checks
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/local-models/trust">
        Open guide
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      Size and price local models
    </td>
    
    <td>
      <a href="https://docs.oz0.ai/fleet/local-models/sizing-and-cost">
        Open guide
      </a>
    </td>
  </tr>
</tbody>
</table>

## Name a local model

```yaml [agents/acme-coder.yaml]
name: acme-coder
runtime: pydantic-ai
model:
  primary: local/qwen3-coder
data: local        # optional: never a cloud model, not even as a fallback
```

`<model>` is the name the server lists at `GET /v1/models`. Ollama lists a model pulled as `qwen3-coder` as `qwen3-coder:latest`; the node serves it under both names, so `local/qwen3-coder` works, while another tag is a name of its own (`local/qwen3-coder:30b`). vLLM lists the name it was started with, such as `Qwen/Qwen3-Coder-30B-A3B-Instruct`, which an agent may write as it is (`local/Qwen/Qwen3-Coder-30B-A3B-Instruct`), or start vLLM with `--served-model-name qwen3-coder` for a shorter one, as below.

With the default placement, an agent on `local/<model>` requires the node label `model.<model>=ready`, so its jobs go only to nodes whose model server serves that model, and only those nodes poll its queue. The runtime calls the model through the node's own proxy, which sends it to the server on loopback; the model call stays on that machine.

With [`model.placement: fleet`](https://docs.oz0.ai/fleet/local-models/fleet#share-a-gpu-with-the-fleet), the job can run on another node. Its model calls carry the conversation to a GPU node through the model's queue, so this placement does not keep the conversation on the job's machine.

With `data: local`, the runtime refuses any cloud model for the agent: `plugin lint` rejects one in `model`, and the runtime refuses to load such an agent. Such an agent names its model as `local/<model>` in `model.primary`, since that is what routes its jobs to the nodes that serve it: `data: local` with an alias (`model.use`) is refused.

Harness agents cannot use `data: local` yet, because a harness runs its own SDK on its own default model.

The edge holds it too: its gateway refuses every model call that names such an agent, whatever sent it (the runtime, a tool's or a hook's own model call, a session's key), with 403 `oz0_data_local`, and the cost log shows the call as refused. Every model the gateway reaches is outside the node, so a call of a `data: local` agent that got there by any path stops there.

An alias may point at a local model too, for example `model alias set coder local/qwen3-coder anthropic/claude-sonnet-5-5`. An alias does not route jobs, though: the job runs on whatever node takes it.

When that node serves the model, it calls its own server. When it does not and a GPU node of the tenant [serves the model to the fleet](https://docs.oz0.ai/fleet/local-models/fleet#share-a-gpu-with-the-fleet), a Pydantic AI agent's model calls go to that model's queue.

Otherwise, and always for a harness agent (its harness calls its own node's proxy), the alias skips the model at once, with the reason `node <id> does not serve qwen3-coder, and no node serves it to the fleet`, and the job runs on the next target. To send an agent's jobs to the nodes that have the model, name it in `model.primary`.

## Configure the node's model servers

The node reads `<data-dir>/models.yaml`, or the file `oz0-node run --models` names:

```yaml [models.yaml]
interval: 10s                 # how often each server is checked
servers:
  - name: ollama
    kind: ollama              # ollama, vllm, llamacpp, mlx, lmstudio or openai (any OpenAI-compatible server)
    url: http://127.0.0.1:11434
    models:                   # only these, each at a digest the server confirms
      - name: qwen3-coder     # or qwen3-coder:latest, as Ollama lists it
        digest: sha256:...    # as GET /api/tags shows it
  - name: vllm
    kind: vllm
    url: http://127.0.0.1:8000
    # The node starts it, on loopback, with a new random key at every start, in VLLM_API_KEY.
    start: [vllm, serve, Qwen/Qwen3-Coder-30B-A3B-Instruct, --served-model-name, qwen3-coder, --host, "{host}", --port, "{port}"]
    allow_unpinned: true      # vLLM reports no digests: serve its models anyway, and say so
    slots: 8                  # optional: at most 8 calls at once; unset, no limit (its parallelism for a fleet server)
    parallel: 16              # optional: how many calls the server runs at once, where it does not say (below)
    formats: [openai-chat, openai-responses]   # optional: the model APIs it serves (below)
    fleet: false              # true: serve its models to the fleet (above), with the node's fleet label
```

`{host}`, `{port}` and `{key}` are replaced when the node starts the server, and the environment has `OZ0_MODEL_HOST`, `OZ0_MODEL_PORT` and `OZ0_MODEL_KEY`. The key is also in the variable the kind reads it from, `VLLM_API_KEY` for vLLM and `LLAMA_API_KEY` for llama.cpp's server, so leave it off the command line: every user on the machine can read a process's arguments with `ps`, and the node warns when `start` has `{key}`.

For another server, read `OZ0_MODEL_KEY` in a wrapper script. An Ollama server the node starts gets `OLLAMA_HOST` on loopback and `OLLAMA_NO_CLOUD=1`. For a server it does not start, `key_env` names an environment variable of the node's that holds its key.

Every `interval`, the node lists each server's models (`GET /v1/models`), and asks mlx-lm, llama.cpp and vLLM for their `/health` too, and reports them in its heartbeats. A server is `ready`, `down` or `refused`, and the reason shows in the node's log, on the Nodes page and in `oz0 node show`.

<table>
<thead>
  <tr>
    <th>
      <code>
        kind
      </code>
    </th>
    
    <th>
      The server
    </th>
    
    <th>
      The node starts it
    </th>
    
    <th>
      Formats it serves
    </th>
    
    <th>
      Its version
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        ollama
      </code>
    </td>
    
    <td>
      Ollama
    </td>
    
    <td>
      When <code>
        start:
      </code>
      
       is given, with <code>
        OLLAMA_HOST
      </code>
      
       on loopback, <code>
        OLLAMA_NO_CLOUD=1
      </code>
      
       and <code>
        OLLAMA_NUM_PARALLEL
      </code>
    </td>
    
    <td>
      Chat Completions, Responses, Messages
    </td>
    
    <td>
      <code>
        GET /api/version
      </code>
      
      ; 0.17.1 or newer
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        vllm
      </code>
    </td>
    
    <td>
      vLLM
    </td>
    
    <td>
      When <code>
        start:
      </code>
      
       is given, its key in <code>
        VLLM_API_KEY
      </code>
    </td>
    
    <td>
      Chat Completions, Responses, Messages
    </td>
    
    <td>
      <code>
        GET /version
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        llamacpp
      </code>
    </td>
    
    <td>
      llama.cpp's <code>
        llama-server
      </code>
    </td>
    
    <td>
      When <code>
        start:
      </code>
      
       is given, its key in <code>
        LLAMA_API_KEY
      </code>
    </td>
    
    <td>
      Chat Completions, Responses, Messages
    </td>
    
    <td>
      <code>
        GET /props
      </code>
      
      's build, such as <code>
        b11457-...
      </code>
      
       (<code>
        min_version: b11000
      </code>
      
       works)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        mlx
      </code>
    </td>
    
    <td>
      mlx-lm's <code>
        mlx_lm.server
      </code>
      
       on Apple silicon
    </td>
    
    <td>
      When <code>
        start:
      </code>
      
       is given: <code>
        mlx_lm.server
      </code>
      
       or <code>
        mlx.launch
      </code>
      
       runs the node's own install (<a href="https://docs.oz0.ai/fleet/local-models-on-a-mac">
        on a Mac
      </a>
      
      ), an absolute path one of yours
    </td>
    
    <td>
      Chat Completions
    </td>
    
    <td>
      Read from the installed packages; mlx 0.29.4 or newer
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        lmstudio
      </code>
    </td>
    
    <td>
      LM Studio's server
    </td>
    
    <td>
      Never: start it yourself; the node reaches it while it runs
    </td>
    
    <td>
      Chat Completions, Responses, Messages
    </td>
    
    <td>
      Not said
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        openai
      </code>
    </td>
    
    <td>
      Any other OpenAI-compatible server
    </td>
    
    <td>
      When <code>
        start:
      </code>
      
       is given, its key in <code>
        OZ0_MODEL_KEY
      </code>
    </td>
    
    <td>
      Chat Completions (list others in <code>
        formats:
      </code>
      
      )
    </td>
    
    <td>
      Not said
    </td>
  </tr>
</tbody>
</table>

### Parallelism

Every server batches calls of its own: it runs several at once, up to its parallelism. The node reports each server's parallelism, and a fleet server's slots follow it unless `slots:` says otherwise. The parallelism is what the server says, else `parallel:` in models.yaml, else the flag of its start command, else its kind's default:

<table>
<thead>
  <tr>
    <th>
      <code>
        kind
      </code>
    </th>
    
    <th>
      Where the node reads it
    </th>
    
    <th>
      Default
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        ollama
      </code>
    </td>
    
    <td>
      <code>
        parallel:
      </code>
      
      ; a server the node starts gets <code>
        OLLAMA_NUM_PARALLEL
      </code>
      
       set to it
    </td>
    
    <td>
      4 when the node starts it (Ollama's own default is 1); 1 when it does not. Ollama keeps memory for every call's context, so set <code>
        parallel: 1
      </code>
      
       for a model that barely fits
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        vllm
      </code>
    </td>
    
    <td>
      <code>
        --max-num-seqs
      </code>
      
       in <code>
        start:
      </code>
    </td>
    
    <td>
      256, vLLM's default on most GPUs
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        llamacpp
      </code>
    </td>
    
    <td>
      <code>
        GET /props
      </code>
      
      's <code>
        total_slots
      </code>
    </td>
    
    <td>
      4: llama.cpp's automatic <code>
        --parallel
      </code>
      
      , whose slots share the whole context. With <code>
        --parallel N
      </code>
      
       it divides <code>
        --ctx-size
      </code>
      
       among the slots unless you add <code>
        --kv-unified
      </code>
      
      , and the node warns
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        mlx
      </code>
    </td>
    
    <td>
      <code>
        --decode-concurrency
      </code>
      
       in <code>
        start:
      </code>
    </td>
    
    <td>
      32, mlx-lm's default; 1 with <code>
        --kv-bits
      </code>
      
      , which turns its batching off
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        lmstudio
      </code>
    </td>
    
    <td>
      <code>
        GET /api/v1/models
      </code>
      
      : each loaded model's Max Concurrent Predictions
    </td>
    
    <td>
      4, LM Studio's default
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        openai
      </code>
    </td>
    
    <td>
      <code>
        parallel:
      </code>
    </td>
    
    <td>
      1
    </td>
  </tr>
</tbody>
</table>

A server that is not a fleet server keeps no limit in the node's proxy unless `slots:` sets one: the server queues what it cannot run at once.

### Formats

`formats:` lists the model APIs a server serves, of those its kind can (the table above); without it, every one its kind serves. The node reports them, and the edge offers the node's local models to a [harness](https://docs.oz0.ai/fleet/local-models-on-a-mac#which-harness-on-which-server) in a format its server serves: Codex speaks Responses and the Claude Agent SDK Messages, so they can use a local model on Ollama, vLLM, llama.cpp or LM Studio, never on mlx-lm. The node's proxy refuses a call in any other format with `404 oz0_local_format`, before the call is metered. Nothing translates one format into another.

### A server that stops answering

A server the node starts that was ready, and then fails three checks in a row (it takes the connection and does not answer within 5 seconds, or its `/health` says it cannot serve), is stopped and started again, with a new key. Its restarts show on the Nodes page and in `oz0 node show`, and `oz0-node doctor` calls such a server `hung`. A server still loading its model, never ready since it started, is left to finish. A server the node does not start gets the status only: restart it yourself.

## Previous section links

<span id="share-a-gpu-with-the-fleet">

Share a GPU with the fleet

</span>

: [open guide](https://docs.oz0.ai/fleet/local-models/fleet#share-a-gpu-with-the-fleet).

<span id="what-the-node-refuses">

What the node refuses

</span>

: [open guide](https://docs.oz0.ai/fleet/local-models/trust#what-the-node-refuses).

<span id="pinned-is-not-vetted">

Pinned is not vetted

</span>

: [open guide](https://docs.oz0.ai/fleet/local-models/trust#pinned-is-not-vetted).

<span id="size-a-machine-from-its-timings">

Size a machine from its timings

</span>

: [open guide](https://docs.oz0.ai/fleet/local-models/sizing-and-cost#size-a-machine-from-its-timings).

<span id="metering">

Metering

</span>

: [open guide](https://docs.oz0.ai/fleet/local-models/sizing-and-cost#metering).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.