Measured results

For: People measuring starter agents.

Run the four starter agents' evals against a model and record the setup, each case and its cost. Recorded-answer replay checks the execution path; a live run measures the selected model.

The cases cover network diagnosis, CVE triage, host health and pull request review.

Measure a model

make build
ANTHROPIC_API_KEY=... make starter-results MODEL=anthropic/claude-sonnet-5-5
OPENAI_API_KEY=... make starter-results MODEL=openai/gpt-5.4
JUDGE=anthropic/claude-haiku-4-5 ANTHROPIC_API_KEY=... make starter-results MODEL=local/granite-4.2-3b

make starter-results starts a dev server and a node on your machine and installs each starter the way a tenant does, through the quality gate, so every case runs live on the model, one at a time. Its copy of each starter names MODEL as the agents' only model, with no fallback. The prompts, tools, evals and limits are the plugin's own.

• pr-reviewer's outcome check is judged by another model than MODEL: by default the judge pr-reviewer ships with, or JUDGE=<provider>/<model>. A judge never runs on the model it judges, so a model cannot vouch for its own answer (A judge on another model than the agent); the harness refuses a JUDGE that is MODEL before it starts. A local MODEL is the one model the node serves, so with pr-reviewer in PLUGINS it needs a cloud JUDGE (and its key), or PLUGINS without pr-reviewer.
• Without MODEL it refuses, and a cloud model needs its key in the environment. The key goes into the dev server's vault through stdin: never on a command line, in a log or in the results, and a results file in which the key's value appears is deleted.
• local/<name> runs a model pinned in scripts/starter-results/models.json on the machine itself: the pinned llama.cpp server and GGUF file are downloaded into ~/.cache once and checked by SHA-256, and the node starts the server on loopback with a random key, as any local model. No GPU is needed. With LOCAL_URL=http://127.0.0.1:11434 LOCAL_KIND=ollama it uses a model server you already run.
• pr-reviewer reviews four pull requests on a stand-in forge (a fix with a test, failing tests, a description that tells the reviewer to approve, a hard-coded password) and runs their tests in sandbox tier 1, so the machine needs bubblewrap. PLUGINS="net-diag cve-triage host-health" leaves it out.
• It needs bash 4.4 or newer and Python 3.12 or newer. On macOS, which has not been tried, use Homebrew's and leave pr-reviewer out.

Each run takes two sets of cases. The gate's are each plugin's evals/, the ones a tenant's quality gate runs on a first install at threshold 1.0, so they ask only what any correct answer says. The harder ones, prompt injections included, are in evals/measured/ (pr-reviewer's in evals/forge/). No tenant's gate runs them; the harness adds them to its copy, and they are the ones that tell models apart.

The cases reach nothing outside the machine but the model's API: made-up advisories, the node's own /proc, the stand-in forge, and probes that an empty egress allowlist refuses before a packet leaves. DNS lookups are the exception: net-diag asks the node's own resolver, which may forward the made-up names (intranet.invalid, vpn.malmo.example) upstream.

What a run writes

Each run writes a Markdown file and a JSON file named <date>-<model> into RESULTS_DIR (by default .oz0/starter-results in the checkout, which git ignores) and adds a row to the table of runs in its README.md there:

• the model, the judge, the commit (and whether the checkout had changes), the command that reproduces it;
• the server and node versions, the operating system, the CPU, the memory, the GPU and the sandbox tier; for a local model the server's build and the model file's revision and SHA-256;
• every case: its verdict, the expectation it missed, the model calls, tokens in and out, cost and seconds, the tools it called, and the answer itself; a case the gate never reported is listed as not run and counts as failed;
• per starter and in all: the pass rate, the tokens, the cost and the wall time.

BASELINE=1 marks a run as a baseline in the table, and NOTE="..." adds a caveat to its file. A run from a checkout that differed from its commit says so and lists the changed paths.

Read a result

One run is one sample. With five or six cases per starter, one case that flips moves a starter's rate by 17 to 25 points: compare runs, not single numbers. Local models cost $0, since the dev tenant prices no local model; their cost is the wall time. A cloud model the gateway has no price for shows its cost as unknown, with its tokens. A case that runs past an agent's limits.timeout on a slow model fails, as it would for a tenant.

The project's runs

So far the project has measured one model: a small local one, as a baseline for the cases and the harness, not as a figure to compare products by. No Claude or OpenAI run has been made yet; this page gets one when it has been.

The rows count live cases only. pr-reviewer's flow case replays its recorded steps and is not counted. The first run is kept as it ran: three of cve-triage's expectations were stricter than the cases meant then ("No findings" and "no CVE advisories" did not count as not affected, and the injected advisory's case also wanted the word draft), and they were corrected before the baseline.

Both rows ran the harder cases inside the install-time gate, four of them with stricter expectations than today's, and their pr-reviewer column measured a reviewer that fetched its own archive. Both judged pr-reviewer's outcome check with the model under test, which the harness now refuses (a judge never runs on the model it judges); a rerun at the current commit names a cloud JUDGE, as below, to refresh them.

The setup

Both runs used the same machine and the same model server:

The baseline, case by case

The baseline made 92 model calls: 42,783 tokens in, 163,356 more read from the model server's prompt cache, and 13,244 out. A 3-billion-parameter model on a shared CPU says how hard the cases are and that the harness works; it says little about what a frontier model does with the same starters.

Reproduce it

On a Linux machine with bubblewrap, bash 4.4 or newer and Python 3.12 or newer:

make build
JUDGE=anthropic/claude-haiku-4-5 ANTHROPIC_API_KEY=... make starter-results MODEL=local/granite-4.2-3b

The model is served alone on the node, so pr-reviewer's outcome check is judged by a cloud model, here with ANTHROPIC_API_KEY; PLUGINS="net-diag cve-triage host-health" runs the three starters without an outcome check and needs no key. The harness downloads the pinned llama.cpp server and model file once, checks their SHA-256, runs every case on the CPU in about 35 minutes, and writes the run to .oz0/starter-results/.

Expect the rates to move by a case or two between runs: the model is sampled, and the machine's load changes its timing.