Run failure drills

For: plugin authors.

Exercise failures and turn written procedures into tested plugin work.

A failure drill

A plugin that writes must survive the node that runs it going down mid-job: a write made twice is a second ticket, a second comment, a second charge. oz0 plugin test --drill proves it on the same dev server and node, against your plugin's own fake of its service and recorded or scripted model answers, never a real service:

oz0 plugin test --drill          # the evals, against the fake, then the drill
oz0 plugin test --drill-only     # the drill alone

For each write it makes, the drill:

1. makes the call as a tool job, through your tenant's policy, hooks and approvals;
2. when the call waits for an approval, crashes the test node's runtime, and approves once the runtime is back: the approval outlives the runtime that asked it. When the crash left the write's outcome unknown and the job waits for a person to say what became of it, the drill answers run_again, so the call is made once more under its key;
3. lets the fake make the write and holds its answer back, crashes the runtime again, and lets the runtime make the call again under the same idempotency key;
4. reads how many writes the fake made.

Each eval case whose job makes a write runs again with the runtime crashed after its step --drill-step (the first by default), and must finish as it did, meeting the case's own expectations. The report says, for each case:

A crash here is the runtime's process group killed, as a crash would end it; the node's supervisor starts it again, and the job goes on from its history. Only a node that dev or plugin test started takes the drill's request, and only while its server is on this machine's loopback: a node of a cluster never does.

The fake and tests/drill.yaml

The drill reads tests/drill.yaml beside your tests. A plugin without one is skipped, with "no fake to drill against", not failed.

# The command that starts your fake, in the plugin's folder; {port} is where it listens, on 127.0.0.1.
fake: [python3, tests/fake_tickets.py, --port, "{port}"]
# How your plugin reaches the fake: {url} is the fake's address. A name your tools read from their environment goes
# in the tool's environment for the drill's install; a secret your plugin declares (a connection's fallback secret
# too) becomes the dev tenant's secret, bound to the fake's address alone.
env:
  TICKETS_URL: "{url}"
# A connection's params in the dev tenant, such as the API address a connection hands each call.
connections:
  tickets:
    api_url: "{url}"
# The writes to make, with their arguments.
writes:
  - function: desk.file_ticket
    arguments: {title: The printer on the third floor is on fire}

Your fake answers GET /__oz0/writes with {"writes": N}: the writes it made, counting a change only once (a second call under a key it has seen made nothing new). The drill reaches the fake through a proxy of its own on loopback, which is how it holds an answer back. With --drill, the evals run against the fake too, so a case whose agent writes reaches it instead of your real service.

The drill points tools that run a server of their own (mcp:, command: or package:) at the fake. A remote server on a connection is your vendor's, and the drill does not stand in for it.

The drill sets a connection's params and its fallback secret, bound to the drill's proxy (a token set by hand goes to an address other than its plugin's own only where it is bound), and walks none of its sign-in steps: a plugin whose calls get their token only from a connection's steps (an OAuth app's sign-in, a token pasted on the edge's page) cannot be drilled end to end yet. Give its fake the hook anyway, so its contract tests count writes the way the drill does.

examples/drill-desk is a small plugin whose write passes; ai.oz0/github carries a tests/drill.yaml too. Both examples currently require repository access; public source downloads are coming soon.

Start from a written procedure

Teams in regulated work often start from a procedure a named expert owns (a claims handler's checklist, a KYC review), not from a prompt. The procedure template makes that procedure the plugin's source of truth:

oz0 plugin new acme-claims --template procedure --description "Answers refund requests for damaged orders"

procedure.md is plain Markdown with a title and four sections:

# Answer a refund request for a damaged order

## Inputs
- The customer's message, with the order number and what is damaged.

## Steps
1. Find the order number and the delivery date in the request.
2. Check that the order was delivered no more than 30 days ago.
3. Recommend a refund of the damaged items' price, never more than the order's total.

## Checks
- The amount you recommend is not more than the order's total.

## When to escalate
- The order was delivered more than 30 days ago.
- The customer threatens legal action or mentions an injury.

The manifest names the procedure, the expert who approved its text, the date they did and how often it is reviewed again:

procedure:
  file: procedure.md
  approved_by: Dana Berg          # a person's name; the template writes "unapproved draft"
  approved_at: "2026-10-12"       # YYYY-MM-DD, not in the future
  review_every: 6mo               # 90d, 26w, 6mo or 1y; a day to three years

oz0 plugin procedure generates, from it, the instructions of each agent whose prompt file does not exist yet or was generated before (prompts/acme-claims.md), and a starter eval set, evals/acme-claims-procedure.yaml: a case that follows the steps, and one per escalation condition, which must answer ESCALATE:. Replace the starter cases' inputs with real cases the expert checked, and delete the eval file's first line to keep your edits: the command rewrites the file only while that line is there. A prompt you wrote by hand is never taken over.

plugin lint checks the procedure: it refuses one without numbered steps, an approver or a valid date, and warns when a section is missing, when the procedure is not approved yet, when its review is due (review due since 2027-04-12), and when an agent's instructions were generated from another text of the procedure (run oz0 plugin procedure again). The generated instructions name neither the approver nor the date: approving the text changes nothing the model reads, and the approver's name is not sent to the model provider.

Once the plugin is installed, the inventory shows its agents' review date and marks it when the date has passed, without anyone running lint again. Orchestrator Zero does not check who the approver is: the field names a person, and the plugin's history shows when the text changed. The procedure tells the agent what to do; the tenant's tool policy, approvals and hooks decide what it may do.