---
title: "Check policy paths"
description: "Check which policy branches your cases exercise."
canonical_url: "https://docs.oz0.ai/improve/evals/coverage"
---
# Check policy paths

> Check which policy branches your cases exercise.

For: Plugin authors testing agents and flows.

Check which policy branches your cases exercise.

## Policy-path coverage

Whether the answers were right is one question; which of the plugin's own policy paths the cases went through is another. A plugin can pass every case while no case ever reaches the refusal of a rule, the denial of an approval, the skipped branch of a flow or a spent budget, the paths that matter when something goes wrong. `--coverage` measures them:

```bash [Terminal]
oz0 plugin test ./support --coverage
oz0 plugin test ./support --min-coverage 80 --coverage-out coverage.json   # either one implies --coverage
```

```text
Policy-path coverage: 5 of 7 branches (71.4%), threshold 60%: ok
KIND      POINT          BRANCH     CASES
rule      rule-ask-e2e   true       covered/approved, covered/denied then quiet
                         false      covered/denied then quiet
rule      rule-run       decided    covered/denied then quiet
approval  greeter.hello  approved   covered/approved
                         denied     covered/denied then quiet
loop      greeter        continued  MISSED
                         stopped    MISSED
```

A **decision point** is a place where the plugin's own policy decides, named `<kind>:<subject>`, with a fixed list of branches. Coverage is the share of all their branches that at least one case reached:

<table>
<thead>
  <tr>
    <th>
      Kind
    </th>
    
    <th>
      Point
    </th>
    
    <th>
      Branches
    </th>
    
    <th>
      Declared by
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        rule
      </code>
    </td>
    
    <td>
      a tool-policy rule, by its ID
    </td>
    
    <td>
      <code>
        decided
      </code>
      
      , or <code>
        true
      </code>
      
       and <code>
        false
      </code>
      
       for a rule with a <code>
        when:
      </code>
    </td>
    
    <td>
      the tool policy an eval ships (below)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        approval
      </code>
    </td>
    
    <td>
      a function (<code>
        support.refund
      </code>
      
      ), or a flow's approval step (<code>
        flow:&lt;flow>/&lt;step>
      </code>
      
      )
    </td>
    
    <td>
      <code>
        approved
      </code>
      
      , <code>
        denied
      </code>
      
      , and <code>
        edited
      </code>
      
       when the tool sets <code>
        approval: {edit: true}
      </code>
    </td>
    
    <td>
      <code>
        requires_approval
      </code>
      
      , the functions the default policy sends to a person by their annotations (destructive, or declaring nothing), the shipped rules that approve, and the functions the taint, origin and sequence rules below may send to a person
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        step
      </code>
    </td>
    
    <td>
      a flow's step (<code>
        &lt;flow>/&lt;step>
      </code>
      
      )
    </td>
    
    <td>
      <code>
        ran
      </code>
      
      , and <code>
        skipped
      </code>
      
       for a step with <code>
        when:
      </code>
    </td>
    
    <td>
      flows
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        loop
      </code>
    </td>
    
    <td>
      an agent's loop rule
    </td>
    
    <td>
      by its <code>
        on_loop
      </code>
      
      : <code>
        reported
      </code>
      
      , or <code>
        continued
      </code>
      
       and <code>
        stopped
      </code>
      
      , or <code>
        stopped
      </code>
    </td>
    
    <td>
      agents with tools
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        budget
      </code>
    </td>
    
    <td>
      an agent's budget
    </td>
    
    <td>
      <code>
        within
      </code>
      
       and <code>
        reached
      </code>
    </td>
    
    <td>
      agents with <code>
        limits.max_cost_usd
      </code>
      
       or <code>
        limits.tree
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        outcome
      </code>
    </td>
    
    <td>
      an agent's outcome check
    </td>
    
    <td>
      <code>
        passed
      </code>
      
       (first round), <code>
        revised
      </code>
      
       (a later round), <code>
        unmet
      </code>
    </td>
    
    <td>
      agents with <code>
        outcome:
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        taint.default
      </code>
    </td>
    
    <td>
      an outbound function
    </td>
    
    <td>
      <code>
        asked
      </code>
      
       (or <code>
        refused
      </code>
      
      , for a floor that refuses) when the job is tainted, <code>
        clear
      </code>
      
       when it is not
    </td>
    
    <td>
      the <a href="https://docs.oz0.ai/govern/taint-and-labels/rules#the-taint-default-rule">
        taint-default rule
      </a>
      
      , when an eval's shipped policy keeps it on
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        origin.check
      </code>
    </td>
    
    <td>
      a sensitive argument (<code>
        &lt;function>/&lt;argument>
      </code>
      
      )
    </td>
    
    <td>
      <code>
        passed
      </code>
      
       or <code>
        below
      </code>
      
       its floor
    </td>
    
    <td>
      arguments with <code>
        x-oz0-role
      </code>
      
       that the schema's <code>
        x-oz0-origin
      </code>
      
       or a shipped origin rule gives a floor (<a href="https://docs.oz0.ai/govern/tool-policy/origins-and-sequences#where-arguments-came-from">
        where arguments came from
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        origin.unknown
      </code>
    </td>
    
    <td>
      an origin rule, by its ID (<code>
        rule-origin-default
      </code>
      
       for the built-in one)
    </td>
    
    <td>
      <code>
        asked
      </code>
      
       (or <code>
        refused
      </code>
      
      ) when an argument is below its floor, <code>
        met
      </code>
      
       when none is
    </td>
    
    <td>
      the built-in origin rule and the shipped origin rules, for the functions they set floors for
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        sequence.match
      </code>
    </td>
    
    <td>
      a sequence rule, by its ID
    </td>
    
    <td>
      <code>
        asked
      </code>
      
       (or <code>
        refused
      </code>
      
      ) after a call that matched its <code>
        after
      </code>
      
      , <code>
        clear
      </code>
      
       when none came before
    </td>
    
    <td>
      the acting sequence rules of a shipped policy
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        tool
      </code>
    </td>
    
    <td>
      a function an agent may call (<code>
        &lt;agent>/&lt;tool>.&lt;function>
      </code>
      
      )
    </td>
    
    <td>
      <code>
        called
      </code>
    </td>
    
    <td>
      each agent's tools; a pattern such as <code>
        crm.*
      </code>
      
       stands for the functions the plugin lists
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        skill
      </code>
    </td>
    
    <td>
      a skill an agent may read (<code>
        &lt;agent>/&lt;plugin>/&lt;skill>
      </code>
      
      )
    </td>
    
    <td>
      <code>
        read
      </code>
    </td>
    
    <td>
      each agent's skills and its tools' manuals
    </td>
  </tr>
</tbody>
</table>

The `tool` and `skill` points are the model's choices rather than the policy's: they show which of an agent's tools no case ever called and which of its skills no case read, and they are counted apart, as `tools and skills chosen: X of Y` after the figure. The policy-path figure, and `--min-coverage`, count the other kinds only, so a threshold a plugin set keeps its meaning. They are read from the cases' steps, as `tools_called` is.

A point a run reached that the plugin does not declare is listed as observed, outside the figure. So is a branch no case can reach in `plugin test`: an approval's `expired` (every approval is answered at once), and `taint.join`, which says whether each tool's result was trusted or tainted the job (what a result carries is its server's to say, not something the plugin's files declare). The edge's intent entry before each call is no decision point: every call that runs has one.

Only the plugin's own policy counts: a tenant's rules are the tenant's, not the plugin's. A plugin that wants its rules tested ships them, as a tool policy an eval names, in the JSON `oz0 tenant tool-policy show --json` prints:

```yaml [evals/refunds.yaml]
agent: support
policy: evals/policy.json    # set on the test tenant while this eval's cases run; the next eval without one gets the default
cases: [...]
```

```json [evals/policy.json]
{"rules": [
  {"id": "rule-big-refund", "match": "support.refund", "action": "approve", "when": "args.amount > 100"},
  {"match": "support.*", "action": "allow"}
]}
```

A rule without an `id` gets one derived from its place and what it says, so it keeps its name from run to run. A rule whose `expires_at` has passed decides nothing, so it is left out, of the policy and of the points. The file must be inside the plugin.

The points come from the plugin as the run leaves it: when `tools.listing.json` is missing, `plugin test` writes the listing its node measured first, and the functions' annotations and schemas are read from that, so a fresh checkout and the next run give the same figure.

How it is recorded: `plugin test --coverage` starts each case's job with an ID that begins with `cov.`, and such a job, with every child job below it, reports each decision point it reaches as a no-op step in its history, the point's name and branch and nothing of what the job read or wrote. The job's trace (`GetJobTrace`) gives them back. Any other job reports nothing.

What it writes:

- **The table**, after the evals' lines, and the whole report under `coverage` in `--json`.
- **`evals/coverage.json`** in the plugin's folder, every point with the cases that reached each branch, which the [studio](https://docs.oz0.ai/build/plugins/studio/test-and-explain#policy-coverage) shows. It is left out of the digest a measured listing records, like `tools.listing.json`.
- **`--coverage-out FILE`**, the same report, for a CI to keep.
- **A Markdown summary** appended to the file `$GITHUB_STEP_SUMMARY` names, when that variable is set: the figure, the counts per kind, the missed branches, and every point in a fold. Any CI that sets the variable gets it; nothing else is assumed.
- **With `--report`**, the [test report](https://docs.oz0.ai/build/plugins/indexes) a listing carries holds the figure and the counts per kind, never the points.

`--min-coverage N` fails the run below N percent and names what no case reached, as the tool-size budget fails it.

[Back to Evals and quality gates](https://docs.oz0.ai/improve/evals).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.