---
title: "Case expectations"
description: "Specify the answer, tool choices, constraints and approval decisions a case must satisfy."
canonical_url: "https://docs.oz0.ai/improve/evals/cases"
---
# Case expectations

> Specify the answer, tool choices, constraints and approval decisions a case must satisfy.

For: Plugin authors testing agents and flows.

Specify the answer, tool choices, constraints and approval decisions a case must satisfy.

## What a case can expect

Every expectation that is set must hold:

<table>
<thead>
  <tr>
    <th>
      Expectation
    </th>
    
    <th>
      Holds when
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        status
      </code>
    </td>
    
    <td>
      The job ends <code>
        completed
      </code>
      
       (the default) or <code>
        failed
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        error_contains
      </code>
    </td>
    
    <td>
      The failed job's error contains this text (with <code>
        status: failed
      </code>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        contains
      </code>
      
      , <code>
        not_contains
      </code>
    </td>
    
    <td>
      The answer, as text (JSON for structured answers), contains each, or none, of these
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        equals
      </code>
    </td>
    
    <td>
      The answer is exactly this value
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        cel
      </code>
    </td>
    
    <td>
      This <a href="https://cel.dev" rel="nofollow">
        CEL
      </a>
      
       expression is true; it sees <code>
        output
      </code>
      
       (the answer), <code>
        cost_usd
      </code>
      
      , <code>
        calls
      </code>
      
       (the job's model calls), <code>
        seconds
      </code>
      
      , <code>
        tool_calls
      </code>
      
       and <code>
        skills_read
      </code>
      
       (<a href="https://docs.oz0.ai/improve/evals/cases#which-tools-and-skills-it-chose">
        below
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        max_cost_usd
      </code>
      
      , <code>
        max_seconds
      </code>
    </td>
    
    <td>
      The job cost and took no more than this
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        tools_called
      </code>
      
      , <code>
        not_called
      </code>
    </td>
    
    <td>
      The case's jobs called each of these functions at least once, or none of them (<a href="https://docs.oz0.ai/improve/evals/cases#which-tools-and-skills-it-chose">
        below
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        skills_read
      </code>
    </td>
    
    <td>
      The case's jobs read each of these skills (<a href="https://docs.oz0.ai/improve/evals/cases#which-tools-and-skills-it-chose">
        below
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        approvals
      </code>
    </td>
    
    <td>
      How a person answers the approvals the case's jobs wait for (below); not a check of its own
    </td>
  </tr>
</tbody>
</table>

An eval passes when the share of passing cases reaches its `threshold`.

### Which tools and skills it chose

A right answer can come the wrong way: from the model's memory instead of the lookup it should have made, or after a delegation that was not needed. A case can say which choices it expects:

```yaml [evals/refunds.yaml]
cases:
  - name: looks the order up before refunding
    input: "Refund order A-1234, it arrived broken"
    expect:
      tools_called: [orders.lookup, payments.refund]
      not_called: [oz.delegate]          # this one needs no specialist
      skills_read: [support/refund-policy]
      cel: "tool_calls.filter(c, c.function == 'refund').size() == 1"
```

- **`tools_called`** and **`not_called`** name functions as `<tool>.<function>`, and the built-ins whose calls are steps of their own: `oz.delegate`, `oz.quarantine`, `oz.skill` and `oz.read_result`. A flow tool is named as its function, `<tool>.<flow>`. The other built-ins run inside the agent's loop and leave no step, so check what they did with `contains` or `cel`.
- **`skills_read`** names skills as `<plugin>/<skill>`: a read with `oz.skill`, or a harness session's own read of its skills folder, which the runtime records the same way.
- **The whole job tree counts**: the case's job and every job it delegated to. So the same case judges an agent that delegates a part and one that does that part itself, which is what a [comparison that folds a delegate in](https://docs.oz0.ai/improve/comparisons-of-a-changed-agent#fold-a-delegate-into-its-parent) needs.
- **In `cel`**, `tool_calls` is the list of the tree's calls in the order they began, each `{tool, function, target, agent}`: `target` is the agent an `oz.delegate` started, `agent` the agent that made the call. `skills_read` is the list of skills read. `calls` stays the job's model calls.

They are read from the steps the jobs' traces record, names only, never what a call carried or returned. A case that checks them fails, with the reason, where its steps cannot be read. `plugin lint` refuses a name the agent cannot reach (its tools, its delegates' tools, its skills), with the nearest one as the fix; where the agent delegates to another plugin's agent, which lint cannot see into, an unknown name is a warning.

### Constraints

A case may start its job with `constraints`, restrictions the job and every job it delegates to must follow, and expect them unchanged at a depth of the job tree with `constraints_reach`: every job down to that depth must carry them word for word (their digest), and at least one job must be that deep.

```yaml [evals/triage.yaml]
cases:
  - name: the EU rule reaches the specialists
    input: "Look into ticket 4711"
    constraints: ["no personal data leaves the EU"]
    expect:
      constraints_reach: 2
```

At most 20 constraints of 500 characters each, one line each; `constraints_reach` is 0 to 8. See [Run receipts](https://docs.oz0.ai/concepts/run-receipts/authority#constraints-that-survive-every-hand-off).

### Approvals in a case

A case never waits for a person who is not there. `plugin test` answers each approval the case's job and its child jobs wait for with the first entry of `expect.approvals` that matches it, as the test tenant's operator:

```yaml [evals/refunds.yaml]
cases:
  - name: a large refund is edited down
    input: Refund order 7 in full
    expect:
      contains: ["refunded 50"]
      approvals:
        - what: support.refund      # the function, or a flow's approval step; * and ? match
          decision: edit            # approve, deny or edit
          edit: { amount: 50 }      # the arguments a decider changes before approving
          comment: half, as agreed
        - kind: loop                # or match by the approval's kind: tool, hook, loop, budget, flow
          decision: deny
```

An approval that no entry matches is denied at once, with "the eval case does not answer this approval", and the case fails for it, however its job ended. A function whose tool asks for a typed confirmation gets the argument's own value. Answers are bound to the arguments the approval shows, as a person's are ([approvals](https://docs.oz0.ai/agents/approvals)).

On your tenant, where the [quality gate](https://docs.oz0.ai/improve/evals/quality-gates#quality-gates) and [eval routines](https://docs.oz0.ai/improve/evals/regressions#scheduled-evals) run the same cases, nothing is ever approved: every approval of a case's jobs is denied at once (by `system:eval`, in the audit log), as the case's `deny` answers say or for want of one, and a case that answers an approval with `approve` or `edit` is skipped there. A skipped case counts neither way, and the result says how many were skipped. Such a case runs in `plugin test`.

[Back to Evals and quality gates](https://docs.oz0.ai/improve/evals).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.