Measure a judge over time

For: Plugin authors checking answer quality.

Use fixed examples to detect judge drift and sample completed jobs for quality measurement.

Anchors: is it the judge or the agent?

A judge is a model too, and the model behind its name can change. When an agent's checks start failing, the agent may have got worse, or the judge may have got stricter. Anchors tell the two apart: a fixed set of examples, a task and an answer each, with the verdict a person gave.

outcome:
  judge: judge
  rubric_file: rubrics/release-notes.md
  anchors_file: anchors/release-notes.yaml

- name: complete
  task: Write the release notes for 1.4.0
  answer: "1.4.0 (2026-10-01): adds exports; breaking: the old flag is gone, use --format instead"
  pass: true
- name: no migration step
  task: Write the release notes for 1.4.0
  answer: "1.4.0: the old flag is gone"
  pass: false
- name: no date
  task: Write the release notes for 1.4.0
  answer: "1.4.0 adds exports"
  pass: false

Three to thirty anchors, each task and answer at most 8 KB; give both verdicts, or a judge that passes everything agrees with them all (lint warns). plugin lint reads the file into the plugin, and the anchors, the rubric and the judge's name together are the set's version: change any, and its history starts again.

At most once a day, when an eval routine runs the agent's evals, the judge judges every anchor again, each as a judge job on a node of the plugin, through the gateway, under the routine's budget: the anchors' runs count toward the routine's daily budget as its cases do.

The anchors cannot change, so the agent cannot be what moved them: when the judge agrees with at least 15 points fewer of them than its first complete run did, and at least two flipped, the judge drifted, and a drift finding with the cause judge says so, naming the anchors that flipped and the model names the provider reported then and now.

oz0 eval anchors lists every check, with its agreement and its baseline's.

Thirty anchors are few for a statistical test: the rule is a threshold with a floor of two flips, which a judge that flips one anchor by chance does not cross. What our tests show is that a stand-in judge that turns its verdicts around is named as the cause and an unchanged one is not.

Sampled judging of finished jobs

An agent without an outcome check has no verdict at all between eval runs. Sampled judging has a judge look at a share of its finished jobs after the fact:

quality:
  sample:
    rate: 0.1               # a tenth of its finished jobs, from 0 to 1
    judge: judge            # an alias or <provider>/<model>
    rubric_file: rubrics/support-reply.md
    anchors_file: anchors/support-reply.yaml   # optional: the judge's anchors, as above
    budget_usd: 0.05        # what one judge job may spend (default 0.05, at most 1)
    daily_budget_usd: 1     # what the agent's judge jobs and its anchors' runs may spend per UTC day (default 1, at most 100)

• Which jobs. Jobs of the agent that completed, that were no replay and no platform job (eval cases, comparisons and judge jobs themselves are never sampled). A job is sampled when a hash of its ID falls under the rate, so every edge agrees and no job is judged twice.
• How. Each sampled job gets a judge job (jdg.s.…) on a node of the plugin: it reads the job's task and answer, as its own sealed input, and makes the same judge call an outcome check makes, through the gateway, metered to the tenant under the agent's name and left out of the usage figures, scorecards and the spend baselines.
  Its verdict is the judge job's answer. The judge job holds a copy of the job it judged, which may be a turn of someone's private chat, so no person opens it: its reasons stay in its history, and the platform reads only the verdict and the cost. Open the judged job itself to see what was said, where you may.
  Since it holds a copy, it starts holding the job's level and compartments, so the walls close it as they close the job, and the gateway holds the judge's call to the tenant's models by level as it held the job's: a judge model the rules do not allow for that level is refused, and the sampled job is recorded as an error.
  A job whose labels cannot be read is not judged.
• What it changes. Nothing about the job: it ended already. The result is a quality signal (the agent's scorecard counts the jobs judged and passed) and, when the judge fails one, a sample_failed finding of the agent with the counts of the last seven days and the latest job; you resolve it. oz0 eval samples lists every sampled job with its verdict, never what it said.
• Jobs about a data subject are sampled only where the tenant allows it, off by default: such a job's task and answer would go to the judge. Turning it on is a governance change that takes a reason and, where the tenant has four eyes on, a second admin; the judge job then runs under the keys of the job's subjects (all of them, when it is about several), follows the tenant's erasure policy as the job did, and is erased as soon as any of them is, even where the agent's keep-for period keeps the job itself longer.
  A job about someone who restricted processing is never sampled, nor one about someone erased after it ran, even where a keep-for period still keeps the job.
• A tenant's own rule wins over the definition, per agent: oz0 tenant quality default --sample support-reply=0.25:judge --reason "..."; rate 0 turns sampling off, and --unsample removes the rule. A rule that names no rubric, for an agent whose definition has no quality.sample to take one from, judges against the default rubric.
  A rule that would sample nothing for an installed agent is refused: one for an agent with an outcome check, one without a judge for an agent whose definition has no quality.sample to take one from, and one with a cloud judge for a data: local agent.
• Not with an outcome check: an agent with outcome: is judged on every job already, and lint refuses quality.sample beside it. A data: local agent's judge must be local/ or test/.
• A judge on the agent's own model still judges: it looks after the fact and blocks nothing, so refusing it would only lose the signal. Its verdict is a weaker one, though, so when a sampled judge job, or a judge on its anchors, ran on the model of the work it judged, the agent gets a judge_same_model finding and the audit log a quality.judge_same_model entry, and the verdict counts as before.
  oz0 tenant quality default --sample warns when a rule's judge is the agent's model now, and keeps the rule.

A sampled judge is a signal for people: it blocks nothing and approves nothing. The one thing its verdicts may do is stop a rollout of a new version that they judge worse than the installed one.

Back to Outcome checks and patterns.