Governance replay

For: tenant admins and governance reviewers.

Replay recently finished jobs to assess a proposed rule, promoted shadow rule, preset or lower budget. The replay decides each recorded job under one change. Model answers and tool results come from its history, so the replay calls no model or tool and spends nothing.

oz0 replay start --promote rule-5c1e9a0d77b2 --since 7d --wait

Replay rpl-01a1...: done
Candidate: promoting rule-5c1e9a0d77b2 from shadow (sha256:7c4f...)

Coverage: 412 listed, 412 read so far; 409 replayed, 0 replay disagrees (left out)
  skipped: 2 erased data subject, 1 unreadable, 0 not replayable
  14 jobs ran harness or remote sessions: their 96 own steps never reach the workflow and were not replayed

Jobs counted: 409; would have gone differently: 37; self-check: 100.0%
Decisions compared before each first change: 2,980; unknown after it (counted neither way): 211

TOOL          JOBS  CHANGED  NOW REFUSED  NOW ASKED  NOW ALLOWED
github.merge  52    37       37           0          0

RULE               CHANGED  NOW REFUSED  NOW ASKED  NOW ALLOWED
rule-5c1e9a0d77b2  37       37           0          0

The tables count each changed job once, at its first change. JOBS is every replayed job of that agent, or every replayed job that called that function or a function of that action class, changed or not: here 37 of the 52 jobs that merged would have been refused. A rule is known only where it changes a decision, so the rule table has no JOBS column. A budget's or a tree limit's change counts under its rule and its agent, not under a function.

What is replayed and what is live

From the record: every model response (the tool calls the model asked for, with their arguments), every tool result, every hook's answer, every person's answer to an approval, and the usage each model call was metered for.

An agent that loads its tools lazily is replayed by the functions its oz.call_tool calls named, as the policy decided them when it ran.

Live, under the change: the tool policy with its rules by caller and kind, its conditions on each call, shadow rules promoted, presets and the plugin's own requires_approval; the taint-default rule over the job's labels at that point; origin rules, against the origins each call's receipt recorded, and sequence rules over the calls before it; grants between plugins; and budgets and tree limits, recomputed from the recorded usage.

The first changed decision

Each tool call a model asked for is a decision point, and so is each model call (for the budgets) and each child job (for the tree's limits). For each job the report gives the first point where the change decides otherwise than the record shows: a call now refused, now sent for approval, now allowed where a person said no or the policy refused, a budget now exceeded, a child beyond the tree's limits.

Everything after that point is unknown. From there the agent would have read another answer and acted differently, so the replay cannot know what came next; those points count neither as changed nor as unchanged.

The self-check

Before the change, each job is replayed under the policy it ran with: the tool policy in force when it started, from the tenant's audit trail, with the label settings in force then (they change apart from the rules). That replay must reproduce every decision the record shows.

A job that does not (a function whose annotations changed since, a grant revoked since, a hook that asked a person) is listed as replay disagrees with its first disagreeing point, and is left out of every count. The report gives the self-check's rate; read the figures only as far as it holds.

When the change promotes a rule or adds a preset, it is applied to what each job ran with. A rule to promote that did not exist yet when a job ran is placed where it stands in the policy now.

Coverage

A report always says what it read and what it did not:

The platform's own jobs (evals, shadow runs, comparisons, test-bench runs) are left out and counted apart, and so are jobs the --agent and --plugin filters leave out.

The window and retention

--since reaches back over finished jobs: 7 days by default, at most --max-jobs jobs (2,000 by default, 10,000 at most), newest first. A replay can only read what Temporal still keeps: a window longer than the namespace's history retention is cut to it, and the report says by how much. Copies kept for a legal hold are never read: a hold keeps evidence; it is not for analysis.

Erasure and content

The edge runs the replay, because it holds the tenant's payload keys: it opens each history in memory, decides it and keeps nothing of it. A report holds counts, job and call IDs, function names, rule IDs and verdicts, never an argument, a result or any text a job read or wrote.

Starting or cancelling a replay takes the governance-write permission and the content permission, since the edge opens what jobs read and wrote on your behalf; reading a report takes governance-read. A reprice's cost figures also take usage-read: without it, a reader sees that the reprice ran, not what anything cost. replay.start, replay.cancel and replay.done (with the coverage) are in the audit log. One replay runs per tenant at a time.

Extra approvals

When a change would send calls to a person, the report counts the extra approvals per day and per decider group: the teams a rule names, or the approvers by role. People a rule names by account are always folded into "other groups", since a rule may name a single person. With works-council mode on, a team below the threshold and a role are folded there too. No figure is given for a person.

The same holds for every other count of approvals. In the tables by agent, function, action class and rule, a change that would ask a group you may not see figures of is moved out of its row into one row, "approvals of other groups", so a rule that names a small team or one person does not show their approvals through its own row. The jobs' rows of those changes are not listed either; the report says how many it left out.

Choosing the change

oz0 replay start --policy stricter.json                  # a whole tool-policy document (JSON)
oz0 replay start --promote rule-5c1e9a0d77b2             # shadow rules turned on
oz0 replay start --preset untrusted-then-consequential   # a shipped preset
oz0 replay start --preset level-l3-writes --preset-rules level.json   # a proposed level change as rules
oz0 replay start --budget-usd 0.50 --budget-children 3   # budget and tree limits per job
oz0 replay show ID --jobs                                # each job's first change
oz0 replay list
oz0 replay cancel ID

A policy document is the stored form, for example:

{"rules": [{"match": "mail.send", "action": "refuse", "when": "!args.to.endsWith('@example.com')"}]}

The Policy page, under Govern, has the same under Replays, and each shadow rule under Policy › Rules has a Replay button that starts one with the rule promoted. A change request that turns a shadow rule on shows that rule's newest replay, if there is one; it never needs one.

What a replay does not tell you

It predicts what the change would have done to the jobs it read, up to each job's first change, and nothing about other jobs or the future. It cannot see past a changed decision, it cannot see what a session's own tools did, and it reads only the window Temporal still keeps. Annotations come from the tools as they are installed now, so a tool whose annotations changed shows up in the self-check rather than as a change.

Reprice recorded spend

A reprice prices the window's recorded model calls again, with no model call and without opening a single payload: it reads the usage record, where each call's tokens are kept (input, output, cache reads and cache writes apart) with what it was charged, effective and at the list price. Give one change:

oz0 replay reprice --prices sheet.json --since 30d --wait      # another price sheet
oz0 replay reprice --model anthropic/claude-haiku-4-5 --wait   # another model's prices on the same tokens
oz0 replay reprice --cache none --wait                          # no prompt cache at all

A price sheet holds a price per model, by <provider>/<model> or by <model>, in dollars per million tokens (and per second, for a model priced by the time it runs):

{"claude-sonnet-5-5": {"input_mtok": 2.5, "output_mtok": 12, "cache_read_mtok": 0.25, "cache_write_mtok": 3.1}}

Each call is priced part by part in micro-dollars, rounded once per part as the usage record rounds, so the figures are exact to the cent and below. A per-second price counts a call's time as the call was charged: from its upstream send, without the time it queued, and none for a call that was refused before it was sent. The report gives the recorded and the repriced cost by kind, effective and at the list price, per agent, per model and per team; a team below the works-council threshold is folded into "other teams".

• Another model's prices (--model) take the tenant's provider record's own price for that model, else the public price list. The report says it every time: same tokens, another price: quality not measured. Whether that model would have done the same work in the same tokens is not something a reprice can tell; a comparison measures it.
• A cache layout: none prices what was read from the prompt cache and written to it as plain input, which says what the cache saved. longer and stable-prefix take the cause the gateway recorded for each cache miss: under longer, misses because the cache had expired read from it instead; under stable-prefix, misses because the tools or the instructions changed do. A call without a recorded cause keeps its recorded cost and is counted. The longer cache's higher write price is not included.

A call without a price for the change, or without tokens (a call charged its reservation because its outcome is unknown), keeps its recorded cost on both sides and is counted.

A call whose cost was recorded as one sum, such as a provider that reports its own cost or a call recorded before costs were split by kind, has that sum replaced by its tokens' new cost, and the report counts these calls: the sum may have held web searches, which then drop out.

Web searches and plugins' units recorded apart are kept as they were, and so is what a call was charged beyond its tokens when its outcome was unknown.

The self-check prices every call again at its own model with today's prices: a call whose price changed since it ran shows there, rather than as a difference.

Your agents' work only: comparisons, evals and other jobs the platform runs about your agents are left out, and so are services plugins report.

Open privilege

An agent's open privilege is what it may call and does not. A privilege replay reads the window's jobs as any replay does and, per agent, sets the functions it could call (the tools it lists, through the grants between plugins and the tool policy) against those the jobs show it called (made, with or without a person's yes).

oz0 replay privilege --since 30d --wait

AGENT    JOBS  COULD CALL  CALLED  SCORE  UNUSED BY CLASS              SUGGESTION
support  412   9           5       52%    destructive 1, egress 2, ... yes

support could call and did not:
  crm.delete_contact (destructive, a person approves, weight 2)
  mail.send_external (egress, weight 4)
  ...

Each function weighs by its action class: destructive and egress 4, a write on the host 3, a write 2, a read 1; a function a person must approve weighs half. The score is the weight of the unused functions over the weight of everything the agent could call: 0 when it used everything it may, 1 when it used none of it.

A rule whose condition reads the call's arguments may let some call through, so it does not take a function away; one on the caller, the tool or the job does. A function the agent lists that no node has listed yet is counted apart and not weighed. The score shows on the agent's scorecard.

With --preset NAME --preset-rules FILE (or --policy, --promote) the score is worked out under a proposed level change instead of the policy now. Such a score describes a policy that is not in force, so the scorecard never shows it.

The tighter candidate

For each agent with enough jobs in the window (--min-jobs, 5 by default), the report suggests a tighter candidate: one rule per unused function that refuses it to that agent alone (when: caller.agent == '<agent>'), with the preset origin privilege. It is a suggestion and is never applied by itself:

oz0 replay privilege suggest ID                         # the rules, as a JSON list of tool rules
oz0 replay privilege test ID --since 30d --wait         # replay them over the same window
oz0 replay privilege propose ID --reason "unused for 30 days"   # add them as shadow rules

test starts a policy replay with the rules as the preset privilege. Rules that refuse only what an agent never called change none of the calls it made; the report shows any call they would have changed, such as one a person was asked about and said no to. propose adds the rules to the tool policy in shadow mode, through the same reviewed path as every tool-policy change (a reason, the audit trail, a second admin where four eyes are on): a shadow rule records what it would have decided and decides nothing until a person promotes it.

Continue at decision points

For a model or a prompt change, a continuation goes one step further than the record: at recorded decision points of finished jobs (model calls whose answer asked for tool calls) it asks the candidate, and beside it the unchanged control, what they would call next, with the record up to that point as context.

oz0 replay continue --model smart-v2 --budget-usd 5 --max-jobs 20 --points 5 -k 3 --since 7d --wait
oz0 replay continue --instructions new-prompt.md --budget-usd 5 --wait

At each point the candidate is asked -k times (3 to 10, 3 by default) and the control -k times. What they propose is decided under the candidate policy (the policy now, or --preset, --policy, --promote for a proposed level change) and never run: no tool is called. The report gives, per point, the share of the candidate's answers that call other functions than the record and than every control answer, and the control's own share against the record; per agent, the divergence: the mean share of the candidate's answers unlike every control answer.

AGENT    POINTS  DIVERGENCE  CANDIDATE VS RECORD  CONTROL VS RECORD  CANDIDATE REFUSED  CONTROL REFUSED
support  100     18%         24%                  7%                 9                  1

Read the candidate against the control: the control's share against the record is the model's own variation, and a candidate is only different by what it adds to that.

A continued job that failed is counted under its failure's kind in a fixed sentence, since what it said could repeat the recorded request. A report from before kept the words: they go only to a reader with the content permission no wall applies to, and anyone else sees that count under one fixed sentence.

These are real model calls. They run as jobs on your nodes (one per continued job, rpl.…), through the gateway, metered, under your budgets and under --budget-usd in all, and show as the platform's own spend, apart from your agents' work.

A job about a data subject is never continued: its content is not sent to a model again. Nor is a job that read data at a level above the lowest or in a compartment, or was started holding such (its labels), or whose labels cannot be read, since the continuation's job carries no labels for the data's rules to hold to; a job that is only untrusted is continued, as nothing proposed is run.

Starting one takes the comparisons-start and content permissions; oz0 replay continue cancel ID stops one and the job it waits on.

What a continuation cannot tell you

It measures proposals at the points it asked, not outcomes: after a point the candidate's next steps would depend on tool results it never got, so nothing after a point is claimed. It does not grade the answers' quality. It asks a model that may belong to another provider than the one the job ran on, with what the job read up to that point: the tenant's data rules still apply to every call. Re-running a whole job from a decision point with live models beside a control fork is not offered yet.