Agents on a Mac with MLX

For: cluster operators.

Set up mlx-lm or LM Studio on an Apple silicon Mac, choose a harness that speaks its API, then configure model prices and routing.

A Mac with Apple silicon runs models well: its memory is shared between the CPU and the GPU, so a 64 GB Mac holds a model that would need a large GPU elsewhere. A node on a Mac serves its models as any local model server does: on loopback, pinned, metered and under the same budgets. This page covers the two servers that are most at home on a Mac, mlx-lm's server and LM Studio, which harness can use which, and how to price and route the work.

Serve a model with mlx-lm

mlx-lm is Apple's MLX library for language models. Its server, mlx_lm.server, speaks OpenAI's Chat Completions. The node installs it itself, the first time a server needs it, from a lock built into the node: every package at an exact version and the hash of its file (mlx-lm 0.32.0 in this release), installed with the node's uv into <data-dir>/tools/mlx-lm/, from PyPI or, on an offline node, from the edge's toolchain. It needs macOS 14 or newer on Apple silicon.

1. Put the model in the Hugging Face cache. The node never downloads a model, and starts mlx-lm offline, so a call naming another model cannot make it download one:
   Terminal

   hf download mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
   

   
   (hf comes with mlx-lm; once the node has installed it, it is in <data-dir>/tools/mlx-lm/*/bin/hf.)
2. Name the server in the node's models.yaml:
   models.yaml

   servers:
     - name: mlx
       kind: mlx
       url: http://127.0.0.1:8080
       # mlx_lm.server is the node's own install; an absolute path runs an install of yours instead.
       start: [mlx_lm.server, --model, mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit, --host, "{host}", --port, "{port}"]
       models:
         - name: mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
           digest: sha256:...   # the model folder's digest (below)
   

3. Restart the node (its service, or oz0-node run), and check it with oz0-node doctor: model server mlx ... ok: serves mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit (mlx 0.32.0; 32 at once; openai-chat). The doctor hashes the model's files to check the pin, which takes a while for a large model; it waits up to 3 minutes, and says it is still hashing after that.

An agent names the model as local/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit, which routes its jobs to the Macs that serve it.

mlx-lm's server checks no key, so it stays on 127.0.0.1, where only users of the Mac reach it; the node refuses one that answers on the Mac's network address.

Its model list names every MLX model in the cache, and a call naming one would load it: the node serves only the models pinned under models:, or, with no pins, only the one the start command names with --model.

A call's body can also name an adapter (adapters) or a draft model (draft_model, num_draft_tokens), which mlx-lm loads from any path or cached model: the node's proxy refuses such a call with 400 oz0_local_field, and a body that names the model a second time in another case (MODEL), which mlx-lm would read differently from the node.

The node also refuses a start: with --draft-model or --adapter-path, whose weights it would neither hash nor check, so a server the node starts serves only the pinned weights.

A server you start yourself runs what its command line names: leave those flags out there too.

mlx-lm says no version over HTTP, so the node reads it from the installed packages, and refuses an mlx older than 0.29.4 (CVE-2025-62608, CVE-2025-62609), and a server whose mlx it cannot read at all.

To use an install of your own, on a Mac or on another machine (mlx has CUDA builds for Linux), give start: the absolute path of its mlx_lm.server: the node reads the versions from that program's environment, following a link such as the one uv tool install mlx-lm or pipx puts in ~/.local/bin. For a server you start yourself, without start:, name its environment with environment:, the folder that python -c 'import sys; print(sys.prefix)' prints in it:

servers:
  - name: mlx
    kind: mlx
    url: http://127.0.0.1:8080
    environment: /Users/me/.local/share/uv/tools/mlx-lm   # where the node reads mlx-lm's and mlx's versions
    models: [{name: mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit, digest: "sha256:..."}]

start: [mlx_lm.server, ...] on a machine other than a Mac with Apple silicon is refused, since the node's own lock is for macOS on Apple silicon.

Pin the model

mlx-lm reports no digest of a model's files, so the node computes one: it hashes the model's folder, each file once, in the background (a large model takes a while the first time, and the server's reason says it is hashing). The digest is the SHA-256 of the files' own SHA-256 lines, sorted by path, which you can compute by hand in the snapshot's folder:

cd ~/.cache/huggingface/hub/models--mlx-community--Qwen3-Coder-30B-A3B-Instruct-4bit/snapshots/*/
find -L . -type f | sed 's|^\./||' | LC_ALL=C sort | xargs shasum -a 256 | shasum -a 256

Or leave the digest out the first time: the server is refused, and its reason on the Nodes page and in oz0-node doctor gives the digest the node found (their files' digests: mlx-community/... sha256:...). A folder with pickled weights (.bin, .pt, .ckpt and the like) is refused. allow_unpinned: true serves the model without a pin, and says so.

How many calls at once

mlx-lm batches calls: it decodes up to 32 at once (--decode-concurrency), and one at a time with a quantized KV cache (--kv-bits). The node reports that number, and a server with fleet: true gets as many slots, so two agents delegated at once both run. parallel: in models.yaml says another number, and slots: caps the node's own proxy. Memory is the real limit on a Mac: each call in flight holds its own context.

Reach LM Studio

LM Studio runs as the person who started it, with the models they downloaded, so the node reaches it while it runs and never starts it (start: is refused). Start its server on 127.0.0.1 (Developer, Start server, or lms server start), leave Serve on Local Network off, and name it:

servers:
  - name: lmstudio
    kind: lmstudio
    url: http://127.0.0.1:1234
    key_env: LMSTUDIO_API_TOKEN   # when Require Authentication is on: a variable of the node's that holds the token
    allow_unpinned: true          # LM Studio reports no digests
    models:
      - name: qwen/qwen3-coder-30b

The node reads LM Studio's REST API for each model's file format (GGUF or MLX) and for the Max Concurrent Predictions of each loaded model (4 by default), which it reports as the server's parallelism. Its proxy passes only inference calls, so LM Studio's own model loading and downloads (/api/v1/models/load, /api/v1/models/download) stay behind, and it refuses a call whose body names a draft_model, which LM Studio would load from any model you downloaded. A default draft model you set in LM Studio's own settings is LM Studio's to load: the node does not see it.

Which harness on which server

The node's proxy never translates one model API into another: an agent runs on a local model only in a format its server serves. The edge knows each server's formats from the node, and offers a harness a local model in one of them.

A server's formats: in models.yaml narrows what it serves, for a server whose Responses or Messages you do not want used.

Harness agents cannot keep their data on the node with data: local yet. A harness agent runs on a local model by its placement: name the model in model.primary, which routes its jobs to the nodes that serve it; its calls then go to the node's own server.

name: mac-coder
runtime: harness:opencode
model:
  primary: local/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit

Price the hardware

A local model costs $0 unless the tenant prices it, so a budget would not see what the Mac costs. Give it a price per million tokens, or per second a call runs, in the tenant's local record:

oz0 provider model local mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit --input 0.05 --output 0.20
oz0 provider model local qwen/qwen3-coder-30b --per-second 0.0002

Every local call is then priced on the job, counted against its budget and shown on the Cost page beside cloud calls, with the node that ran it.

Local first, the cloud when a round fails

A cascade runs a round on a local model first, and restarts it on a larger model only when its answer fails the agent's check:

model:
  primary: local/mlx-community/Qwen3-Coder-30B-A3B-Instruct-4bit
  cascade: [anthropic/claude-sonnet-5-5]   # only when the local round fails
outcome:
  grader:
    contains: ["## Summary"]

• An agent with data: local names only local/ models in its cascade: every round stays on the node, and nothing escalates to the cloud. Escalating to a cloud model is for agents without it.
• The nodes that run a cascading agent must serve every local/ model it names, since a restarted round runs on the job's node.
• A cascade does not go with model.placement: fleet, so it cannot use a model another Mac serves to the fleet.

Stable prompts keep the local cache warm

mlx-lm keeps the KV cache of recent prompts and reuses the longest prefix a new call shares with one of them, as llama.cpp's slots, vLLM's prefix cache and Ollama do.

The runtime keeps each prompt's stable parts first: instructions, the skills index, tool definitions in a fixed order, then the history, with the status block and fresh results last (prompts with their stable parts first).

A long agent's next call then shares all but its last part with the one before, and the server reads it from its cache instead of computing it again. Keep your own instructions free of times and counters for the same reason.

A Mac Studio cluster over Thunderbolt

mlx can spread one model across several Macs connected with Thunderbolt cables (mlx.launch with its ring backend, or jaccl for RDMA over Thunderbolt 5), and mlx-lm's server runs on such a group: every Mac loads its part of the model, and only the first (rank 0) answers HTTP. Such a group can be a fleet model server, with the first Mac as its node:

1. Connect the Macs with Thunderbolt and let mlx find the links: mlx.distributed_config --over thunderbolt --hosts mac1,mac2 --output-hostfile ~/hosts.json (it may ask for sudo to set up the links). The first host in the file is rank 0: make it the node's Mac.
2. Install the same mlx-lm at the same path on every Mac, for example from the lock the node installs its own from: once the node has started an mlx_lm.server of its own (in this guide), it is in <data-dir>/tools/mlx-lm/<id>/requirements.txt on the node's Mac. Copy it to each Mac and run uv venv ~/mlx-lm and then uv pip install --require-hashes --no-deps --python ~/mlx-lm/bin/python -r requirements.txt there. Download the model into each Mac's cache, and let the node's user reach the others with ssh.
3. Start the group from the node:
   models.yaml

   servers:
     - name: studio-cluster
       kind: mlx
       url: http://127.0.0.1:8080
       start: [/Users/me/mlx-lm/bin/mlx.launch, --hostfile, /Users/me/hosts.json, --backend, ring, --env, HF_HUB_OFFLINE=1, --,
               /Users/me/mlx-lm/bin/mlx_lm.server, --model, mlx-community/Qwen3-Coder-480B-A35B-Instruct-4bit,
               --host, "{host}", --port, "{port}"]
       models: [{name: mlx-community/Qwen3-Coder-480B-A35B-Instruct-4bit, digest: "sha256:..."}]
       fleet: true
   

   
   and give the node the fleet label on its join token (oz0 token create --label fleet=true) or when accepting it, as for any fleet server.

How it meets the node's checks, and what it does not do yet:

• The HTTP server listens on 127.0.0.1 of the first Mac only, so the node's check that a server answers on no network address passes. The Macs' own links between ranks use other ports on the Thunderbolt interfaces, which the node does not check: anything on those cables can reach them, so connect only the cluster's Macs.
• The node checks, pins and hashes the first Mac's copy of the model, and reads the versions of the first Mac's mlx-lm. It cannot see the other Macs' copies or versions: keep them the same by hand.
• The node starts and stops mlx.launch on its own Mac; whether the other Macs' processes stop with it is up to mlx's launcher. A hung group is started again like any server the node starts.
• mlx.launch runs the program after -- at the same path on every Mac, so give it by its absolute path, as above. With the node's own install (start: [mlx.launch, ..., --, mlx_lm.server, ...]), the node gives both programs by their paths in <data-dir>/tools/mlx-lm/, and the other Macs need the same install at that path.
• Only the node's Mac is a node: the other Macs run no jobs and report nothing to the cluster.