Configure local models

For: cluster operators.

A model server on a node is a capability of that node. A laptop with Ollama, a Mac with MLX or LM Studio (Agents on a Mac), or a GPU box with vLLM, becomes a node that agents on local models are routed to, under the same budgets and in the same cost log as cloud models.

Choose a task

Name a local model

name: acme-coder
runtime: pydantic-ai
model:
  primary: local/qwen3-coder
data: local        # optional: never a cloud model, not even as a fallback

<model> is the name the server lists at GET /v1/models. Ollama lists a model pulled as qwen3-coder as qwen3-coder:latest; the node serves it under both names, so local/qwen3-coder works, while another tag is a name of its own (local/qwen3-coder:30b). vLLM lists the name it was started with, such as Qwen/Qwen3-Coder-30B-A3B-Instruct, which an agent may write as it is (local/Qwen/Qwen3-Coder-30B-A3B-Instruct), or start vLLM with --served-model-name qwen3-coder for a shorter one, as below.

With the default placement, an agent on local/<model> requires the node label model.<model>=ready, so its jobs go only to nodes whose model server serves that model, and only those nodes poll its queue. The runtime calls the model through the node's own proxy, which sends it to the server on loopback; the model call stays on that machine.

With model.placement: fleet, the job can run on another node. Its model calls carry the conversation to a GPU node through the model's queue, so this placement does not keep the conversation on the job's machine.

With data: local, the runtime refuses any cloud model for the agent: plugin lint rejects one in model, and the runtime refuses to load such an agent. Such an agent names its model as local/<model> in model.primary, since that is what routes its jobs to the nodes that serve it: data: local with an alias (model.use) is refused.

Harness agents cannot use data: local yet, because a harness runs its own SDK on its own default model.

The edge holds it too: its gateway refuses every model call that names such an agent, whatever sent it (the runtime, a tool's or a hook's own model call, a session's key), with 403 oz0_data_local, and the cost log shows the call as refused. Every model the gateway reaches is outside the node, so a call of a data: local agent that got there by any path stops there.

An alias may point at a local model too, for example model alias set coder local/qwen3-coder anthropic/claude-sonnet-5-5. An alias does not route jobs, though: the job runs on whatever node takes it.

When that node serves the model, it calls its own server. When it does not and a GPU node of the tenant serves the model to the fleet, a Pydantic AI agent's model calls go to that model's queue.

Otherwise, and always for a harness agent (its harness calls its own node's proxy), the alias skips the model at once, with the reason node <id> does not serve qwen3-coder, and no node serves it to the fleet, and the job runs on the next target. To send an agent's jobs to the nodes that have the model, name it in model.primary.

Configure the node's model servers

The node reads <data-dir>/models.yaml, or the file oz0-node run --models names:

interval: 10s                 # how often each server is checked
servers:
  - name: ollama
    kind: ollama              # ollama, vllm, llamacpp, mlx, lmstudio or openai (any OpenAI-compatible server)
    url: http://127.0.0.1:11434
    models:                   # only these, each at a digest the server confirms
      - name: qwen3-coder     # or qwen3-coder:latest, as Ollama lists it
        digest: sha256:...    # as GET /api/tags shows it
  - name: vllm
    kind: vllm
    url: http://127.0.0.1:8000
    # The node starts it, on loopback, with a new random key at every start, in VLLM_API_KEY.
    start: [vllm, serve, Qwen/Qwen3-Coder-30B-A3B-Instruct, --served-model-name, qwen3-coder, --host, "{host}", --port, "{port}"]
    allow_unpinned: true      # vLLM reports no digests: serve its models anyway, and say so
    slots: 8                  # optional: at most 8 calls at once; unset, no limit (its parallelism for a fleet server)
    parallel: 16              # optional: how many calls the server runs at once, where it does not say (below)
    formats: [openai-chat, openai-responses]   # optional: the model APIs it serves (below)
    fleet: false              # true: serve its models to the fleet (above), with the node's fleet label

{host}, {port} and {key} are replaced when the node starts the server, and the environment has OZ0_MODEL_HOST, OZ0_MODEL_PORT and OZ0_MODEL_KEY. The key is also in the variable the kind reads it from, VLLM_API_KEY for vLLM and LLAMA_API_KEY for llama.cpp's server, so leave it off the command line: every user on the machine can read a process's arguments with ps, and the node warns when start has {key}.

For another server, read OZ0_MODEL_KEY in a wrapper script. An Ollama server the node starts gets OLLAMA_HOST on loopback and OLLAMA_NO_CLOUD=1. For a server it does not start, key_env names an environment variable of the node's that holds its key.

Every interval, the node lists each server's models (GET /v1/models), and asks mlx-lm, llama.cpp and vLLM for their /health too, and reports them in its heartbeats. A server is ready, down or refused, and the reason shows in the node's log, on the Nodes page and in oz0 node show.

Parallelism

Every server batches calls of its own: it runs several at once, up to its parallelism. The node reports each server's parallelism, and a fleet server's slots follow it unless slots: says otherwise. The parallelism is what the server says, else parallel: in models.yaml, else the flag of its start command, else its kind's default:

A server that is not a fleet server keeps no limit in the node's proxy unless slots: sets one: the server queues what it cannot run at once.

Formats

formats: lists the model APIs a server serves, of those its kind can (the table above); without it, every one its kind serves. The node reports them, and the edge offers the node's local models to a harness in a format its server serves: Codex speaks Responses and the Claude Agent SDK Messages, so they can use a local model on Ollama, vLLM, llama.cpp or LM Studio, never on mlx-lm. The node's proxy refuses a call in any other format with 404 oz0_local_format, before the call is metered. Nothing translates one format into another.

A server that stops answering

A server the node starts that was ready, and then fails three checks in a row (it takes the connection and does not answer within 5 seconds, or its /health says it cannot serve), is stopped and started again, with a new key. Its restarts show on the Nodes page and in oz0 node show, and oz0-node doctor calls such a server hung. A server still loading its model, never ready since it started, is left to finish. A server the node does not start gets the status only: restart it yourself.

Previous section links

Share a GPU with the fleet: open guide.

What the node refuses: open guide.

Pinned is not vetted: open guide.

Size a machine from its timings: open guide.

Metering: open guide.