--- title: "Record and replay answers" description: "Record model answers and flow outputs, then reuse them in repeatable tests." canonical_url: "https://docs.oz0.ai/improve/evals/recordings" --- # Record and replay answers > Record model answers and flow outputs, then reuse them in repeatable tests. For: Plugin authors testing agents and flows. Record model answers and flow outputs, then reuse them in repeatable tests. ## Record once, replay after Every model call of a case goes through the test server's gateway, and from there through a recorder in the CLI: - **`--record`** calls the real API for every case, with `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` from your environment, and keeps each answer in `evals/cassettes//.jsonl`. - **Without flags**, a case that has a recording is answered from it, in order, without calling anyone and without a key; a case without one calls the real API. Calls that run at the same time, such as a [best-of's](https://docs.oz0.ai/improve/outcome-checks/selection#best-of-n) candidates, each get the recorded call with the same request, whatever order they arrive in. - **`--live`** ignores recordings. Commit the recordings with the plugin: they hold what the model answered, never your key or the request's headers. When a prompt changes, a replay still answers in order and warns that the call differs from the recording; record again to bring it up to date. The comparison leaves out the [status block](https://docs.oz0.ai/build/parts/contracts-and-limits/budgets#what-the-agent-sees-of-its-spend) the runtime ends each request of a loop with, since its time and budget differ from run to run. A case that makes more calls than its recording holds fails and says so. Costs in a replay are what the recorded calls cost. The studio's [test bench](https://docs.oz0.ai/build/plugins/studio/test-and-explain#the-test-bench) replays the same cassettes against the installed plugin, on the cluster's own nodes: there the edge answers the job's model calls from the recording, and they cost nothing. An eval case has no rubric of its own for another model to score. An agent with an [outcome check](https://docs.oz0.ai/improve/outcome-checks/rollouts#tests-and-the-quality-gate) runs it in every case, recorded with the rest, so a case can expect it to pass or to fail. Running cases in parallel comes later. ## Recorded step outputs of flows A flow's case can go one step further: record what each of the flow's steps returned once, and replay that instead of calling the flow's tools, agents and approvals. The flow's own logic still runs on the recorded values, its `when:` conditions, the `${{ }}` expressions in each step's input and its `output:`, so the case checks the flow and has no side effects: no tool writes anything, no agent spends a token, nobody is asked to approve. ```bash [Terminal] oz0 plugin test ./support --record-steps # run the flows' cases for real, keep each step's output oz0 plugin test ./support # replay them ``` ```text PASS triage / a double charge reaches billing (recorded steps, 3 calls, $0.0021, 4.1 s) recorded 3 steps in evals/steps/triage/a-double-charge-reaches-billing.json ``` A recording lives next to the eval, in `steps//.json` (names lowercased, other characters as `-`), and is plain JSON you can review and edit: ```json [evals/steps/triage/a-double-charge-reaches-billing.json] { "format": "oz0.flow-steps/v1", "flow": "jev-triage", "steps": [ { "id": "decide", "kind": "tool", "name": "jev.decide", "input": { "state": "..." }, "output": { "team": { "choice": "billing", "confidence": 0.93 } } }, { "id": "route", "kind": "agent", "name": "jev-billing", "input": "...", "output": "Refunded the duplicate charge." } ] } ``` Only the steps that ran are listed, in order. A step can have an `error` instead of an `output`, and the replay fails it with that error. `uses` lists the other plugins the flow's plugin could reach when it was recorded; the replay's expressions read them as the flow's [`uses`](https://docs.oz0.ai/build/parts/flows/steps-and-expressions#steps-of-an-optional-plugin) instead of the tenant's grants, since a replay calls nothing. So a flow that works with whichever of several systems a tenant has gets a case per system, each with its own `uses`, and every case runs at the quality gate of a tenant that has only one of them, or none. `plugin lint` checks every recording against its flow: a step that is not in the flow, or whose kind changed, is an error. When a replay reaches a step the recording has no output for, as when you changed a `when:` so that another step runs, the case fails and says to record it again. The quality gate replays recordings too: a flow's case with a recording runs on the test node without calling the flow's agents or tools, while cases without one run for real. The studio's test bench records and replays the same way, through the runtime API's `record_steps` and `replay_steps` on `StartJob` (a finished job's recording is `Job.recorded_steps`). A replay chooses every step's outcome, so only the tenant's admins and owners may start one; it is marked as a replay (memo `oz0.replay`), kept out of counts and rollouts' gates, and recorded as `run.replay` in the audit log. A recorded case's verdict at the gate rests on the outputs in your plugin's files, so review recordings as you review code. Recordings stay in the plugin's files and do not travel to nodes; the gate reads them from the installed artifact, and a case whose recording cannot be read fails rather than run for real. A finished job's recording is content: only callers with the content permission see it. [Back to Evals and quality gates](https://docs.oz0.ai/improve/evals). ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.