Check policy paths

For: Plugin authors testing agents and flows.

Check which policy branches your cases exercise.

Policy-path coverage

Whether the answers were right is one question; which of the plugin's own policy paths the cases went through is another. A plugin can pass every case while no case ever reaches the refusal of a rule, the denial of an approval, the skipped branch of a flow or a spent budget, the paths that matter when something goes wrong. --coverage measures them:

oz0 plugin test ./support --coverage
oz0 plugin test ./support --min-coverage 80 --coverage-out coverage.json   # either one implies --coverage

Policy-path coverage: 5 of 7 branches (71.4%), threshold 60%: ok
KIND      POINT          BRANCH     CASES
rule      rule-ask-e2e   true       covered/approved, covered/denied then quiet
                         false      covered/denied then quiet
rule      rule-run       decided    covered/denied then quiet
approval  greeter.hello  approved   covered/approved
                         denied     covered/denied then quiet
loop      greeter        continued  MISSED
                         stopped    MISSED

A decision point is a place where the plugin's own policy decides, named <kind>:<subject>, with a fixed list of branches. Coverage is the share of all their branches that at least one case reached:

The tool and skill points are the model's choices rather than the policy's: they show which of an agent's tools no case ever called and which of its skills no case read, and they are counted apart, as tools and skills chosen: X of Y after the figure. The policy-path figure, and --min-coverage, count the other kinds only, so a threshold a plugin set keeps its meaning. They are read from the cases' steps, as tools_called is.

A point a run reached that the plugin does not declare is listed as observed, outside the figure. So is a branch no case can reach in plugin test: an approval's expired (every approval is answered at once), and taint.join, which says whether each tool's result was trusted or tainted the job (what a result carries is its server's to say, not something the plugin's files declare). The edge's intent entry before each call is no decision point: every call that runs has one.

Only the plugin's own policy counts: a tenant's rules are the tenant's, not the plugin's. A plugin that wants its rules tested ships them, as a tool policy an eval names, in the JSON oz0 tenant tool-policy show --json prints:

agent: support
policy: evals/policy.json    # set on the test tenant while this eval's cases run; the next eval without one gets the default
cases: [...]

{"rules": [
  {"id": "rule-big-refund", "match": "support.refund", "action": "approve", "when": "args.amount > 100"},
  {"match": "support.*", "action": "allow"}
]}

A rule without an id gets one derived from its place and what it says, so it keeps its name from run to run. A rule whose expires_at has passed decides nothing, so it is left out, of the policy and of the points. The file must be inside the plugin.

The points come from the plugin as the run leaves it: when tools.listing.json is missing, plugin test writes the listing its node measured first, and the functions' annotations and schemas are read from that, so a fresh checkout and the next run give the same figure.

How it is recorded: plugin test --coverage starts each case's job with an ID that begins with cov., and such a job, with every child job below it, reports each decision point it reaches as a no-op step in its history, the point's name and branch and nothing of what the job read or wrote. The job's trace (GetJobTrace) gives them back. Any other job reports nothing.

What it writes:

• The table, after the evals' lines, and the whole report under coverage in --json.
• evals/coverage.json in the plugin's folder, every point with the cases that reached each branch, which the studio shows. It is left out of the digest a measured listing records, like tools.listing.json.
• --coverage-out FILE, the same report, for a CI to keep.
• A Markdown summary appended to the file $GITHUB_STEP_SUMMARY names, when that variable is set: the figure, the counts per kind, the missed branches, and every point in a fold. Any CI that sets the variable gets it; nothing else is assumed.
• With --report, the test report a listing carries holds the figure and the counts per kind, never the points.

--min-coverage N fails the run below N percent and names what no case reached, as the tool-size budget fails it.

Back to Evals and quality gates.