Case expectations

For: Plugin authors testing agents and flows.

Specify the answer, tool choices, constraints and approval decisions a case must satisfy.

What a case can expect

Every expectation that is set must hold:

An eval passes when the share of passing cases reaches its threshold.

Which tools and skills it chose

A right answer can come the wrong way: from the model's memory instead of the lookup it should have made, or after a delegation that was not needed. A case can say which choices it expects:

cases:
  - name: looks the order up before refunding
    input: "Refund order A-1234, it arrived broken"
    expect:
      tools_called: [orders.lookup, payments.refund]
      not_called: [oz.delegate]          # this one needs no specialist
      skills_read: [support/refund-policy]
      cel: "tool_calls.filter(c, c.function == 'refund').size() == 1"

• tools_called and not_called name functions as <tool>.<function>, and the built-ins whose calls are steps of their own: oz.delegate, oz.quarantine, oz.skill and oz.read_result. A flow tool is named as its function, <tool>.<flow>. The other built-ins run inside the agent's loop and leave no step, so check what they did with contains or cel.
• skills_read names skills as <plugin>/<skill>: a read with oz.skill, or a harness session's own read of its skills folder, which the runtime records the same way.
• The whole job tree counts: the case's job and every job it delegated to. So the same case judges an agent that delegates a part and one that does that part itself, which is what a comparison that folds a delegate in needs.
• In cel, tool_calls is the list of the tree's calls in the order they began, each {tool, function, target, agent}: target is the agent an oz.delegate started, agent the agent that made the call. skills_read is the list of skills read. calls stays the job's model calls.

They are read from the steps the jobs' traces record, names only, never what a call carried or returned. A case that checks them fails, with the reason, where its steps cannot be read. plugin lint refuses a name the agent cannot reach (its tools, its delegates' tools, its skills), with the nearest one as the fix; where the agent delegates to another plugin's agent, which lint cannot see into, an unknown name is a warning.

Constraints

A case may start its job with constraints, restrictions the job and every job it delegates to must follow, and expect them unchanged at a depth of the job tree with constraints_reach: every job down to that depth must carry them word for word (their digest), and at least one job must be that deep.

cases:
  - name: the EU rule reaches the specialists
    input: "Look into ticket 4711"
    constraints: ["no personal data leaves the EU"]
    expect:
      constraints_reach: 2

At most 20 constraints of 500 characters each, one line each; constraints_reach is 0 to 8. See Run receipts.

Approvals in a case

A case never waits for a person who is not there. plugin test answers each approval the case's job and its child jobs wait for with the first entry of expect.approvals that matches it, as the test tenant's operator:

cases:
  - name: a large refund is edited down
    input: Refund order 7 in full
    expect:
      contains: ["refunded 50"]
      approvals:
        - what: support.refund      # the function, or a flow's approval step; * and ? match
          decision: edit            # approve, deny or edit
          edit: { amount: 50 }      # the arguments a decider changes before approving
          comment: half, as agreed
        - kind: loop                # or match by the approval's kind: tool, hook, loop, budget, flow
          decision: deny

An approval that no entry matches is denied at once, with "the eval case does not answer this approval", and the case fails for it, however its job ended. A function whose tool asks for a typed confirmation gets the argument's own value. Answers are bound to the arguments the approval shows, as a person's are (approvals).

On your tenant, where the quality gate and eval routines run the same cases, nothing is ever approved: every approval of a case's jobs is denied at once (by system:eval, in the audit log), as the case's deny answers say or for want of one, and a case that answers an approval with approve or edit is skipped there. A skipped case counts neither way, and the result says how many were skipped. Such a case runs in plugin test.

Back to Evals and quality gates.