--- title: "Comparisons of a changed agent" description: "Run an agent as installed against a studio draft, another revision of its plugin, the agent without one of its skills or with a delegate folded in, on the same cases, side-effect free, and see whether the change is better, the same or worse." canonical_url: "https://docs.oz0.ai/improve/comparisons-of-a-changed-agent" --- # Comparisons of a changed agent > Run an agent as installed against a studio draft, another revision of its plugin, the agent without one of its skills or with a delegate folded in, on the same cases, side-effect free, and see whether the change is better, the same or worse. For: Plugin authors evaluating a change. Compare the installed agent with one changed definition before saving it. The comparison runs both versions on the same eval cases as shadow runs, then reports whether their paired pass rates make the variant better, the same or worse. This comparison measures quality. Use [measured savings](https://docs.oz0.ai/improve/measured-savings) for a cost comparison. ## What the variant can be
Variant oz0 compare start AGENT --variant ... Runs on
Your studio draft of the agent's plugin draft=<folder> a private candidate of the plugin, on the nodes --nodes names, while the comparison lives
A private candidate of the plugin revision=candidate:<id> the candidate's own nodes (its owner or a tenant admin may compare it)
The candidate the plugin's quality gate tests revision=gate the gate's test node
The candidate of the plugin's rollout revision=rollout the rollout's nodes
A saved revision of the plugin's workspace folder revision=workspace:<revision> a private candidate, on the nodes --nodes names
A version of the plugin the tenant had revision=version:<version> a private candidate, from its stored artifact, on the nodes --nodes names
The agent without one of its skills withhold-skill=<skill> the installed version: the skill is left out of the agent's skills index and its reads, in that arm only
The agent with one of its delegates folded in inline-delegate=<agent> the installed version: the delegate's instructions become a skill and its tools join the agent's , in that arm only
The baseline is always the installed version as defined, and the variant changes one thing. Agents on any runtime may be compared: Pydantic AI agents, OpenAI Agents SDK agents and harness agents such as the Claude Agent SDK. A `best_of:` agent may not, since its candidates are jobs of their own. A draft, a workspace revision and a past version are placed as a **private candidate** of the plugin for the comparison's life, on the nodes with the labels `--nodes` names (an operator's only on nodes open to candidates, with the secrets the installed version gets, as [a developer's candidate](https://docs.oz0.ai/build/plugins/develop-and-test/candidate-checks#try-it-on-a-real-cluster)). Those nodes take no other work of the tenant meanwhile, so name nodes that leave the installed version others. Since that places code on your nodes, starting such a comparison needs what making a private candidate needs: the `candidates.write` permission and, unless you are a tenant admin, the content permission, because the candidate's code runs next to the node's other plugins and sees what the jobs it serves read and wrote. The server renews the candidate while the comparison waits or runs and removes it within seconds of the comparison's end; both are in the audit log. The studio's test bench still runs only the installed version. ## Fold a delegate into its parent Whether a split into a parent and a delegate pays is a question of quality, and a comparison answers it on the parent's own cases. `inline-delegate=` runs the parent, in the variant arm only, as one agent that does the delegate's part itself: - **the delegate's instructions become a skill of the parent**, named after the delegate, listed in its skills index and read with `oz.skill` as any skill is; - **the delegate's tools and skills join the parent's**, and its own delegates join the parent's other delegates; - **`oz.delegate` to that delegate is withheld**: the arm is not offered it, and the edge refuses the arm's child job of that agent too. The parent's calls of the delegate's tools are its own, under the same grants and the same tool policy as any of its calls, and the arm is a shadow run as every comparison's run is. The fold is judged as the agent it builds: what the tenant's policy says of the delegate's calls still holds for the calls the parent makes in its place. The delegate must be an agent of the parent's own plugin that the parent may delegate to, both on Pydantic AI, neither a `best_of:` agent, and the plugin must have no skill of the delegate's name. A delegate of another plugin is that plugin's to change, so it is not folded in. A split the tenant's policy keeps is not folded, and the comparison is refused with the reason: - **Data that stays on the node.** A delegate with `data: local` folds only into a parent with `data: local`: the folded agent's model is the parent's, and it reads the delegate's tools' results. - **Clearance.** A delegate with a [clearance](https://docs.oz0.ai/govern/taint-and-labels/label-policies#label-policies) (the tenant's rule for it, or its definition's) folds only into a parent whose clearance stays within it: no higher level and no compartment the delegate lacks. Otherwise the delegate's tools' results would reach a model cleared for more. - **Duty pairs.** When the folded agent would hold both sides of one of the tenant's [duty pairs](https://docs.oz0.ai/govern/tool-policy/duties#duties) that the parent does not hold alone, the fold is refused: the parent and the delegate hold the two sides between them, or the delegate holds both itself (the rule checks those calls as the delegate's, and folded in they would be the parent's).
Either way the split keeps two duties apart, which is a decision about separation of duties, not a question a comparison of quality can settle. Change the duty rules first if the split is no longer wanted. - **Hooks and hidden skills for the delegate alone.** When a plugin's hook names the delegate in its `agents:` and not the parent, or the tenant hides a skill from the delegate that it shows the parent, the arm is refused when its job starts: folded in, the delegate's tool calls and skills are the parent's, so that hook would not be asked about them and that skill would be in the parent's index. - **Rules by the calling agent.** When a rule, floor or gate of the tool policy has a condition that reads the calling agent (`caller.agent` in any form) and may decide a function only the delegate's tools reach, the fold is refused: folded in, the delegate's calls of that function are made as the parent, so the rule would no longer decide them as it decides the delegate's.
Whether a condition treats two agents alike cannot be told without the calls' arguments, so any condition that reads the agent counts, a rule written for both agents too. A command rule for the delegate's jobs and not the parent's is refused the same way. Compare without the fold, or write such a rule without the calling agent first. These are checked when the comparison starts and again on the edge before each run of it, so a duty rule, a rule by agent or a clearance added while it waits for a node stops it with the reason. While it runs, the edge judges each of the arm's calls as the folded agent's: a call that completes a duty pair the parent does not hold alone is refused, with no `duty_pair` finding about the parent for it (the arm is not the agent as installed), and a call of a function only the delegate's tools reach is decided as the delegate's call too, so it is refused when the policy refuses it to the delegate. The verdict says better, same or worse as for any change, and lists the [cases that flipped](#the-verdict). Write cases that say which tools and skills they expect (`tools_called`, `not_called`, `skills_read`, [Evals](https://docs.oz0.ai/improve/evals/cases#which-tools-and-skills-it-chose)): they count the whole job tree, so the same case holds of the parent that delegates and the one that does the work itself. The findings `delegate_every_job` and `delegate_chain` ([What a delegation hands over](https://docs.oz0.ai/agents/delegation#what-a-delegation-hands-over)) name parents worth trying this on. ```bash oz0 compare start lead --variant inline-delegate=researcher --budget 2 --wait ``` ## Nothing acts on the world Every run is a [shadow run](https://docs.oz0.ai/build/plugins/install/rollouts#shadow-runs-for-a-quiet-tenant): a function the tenant's policy does not list as read-only answers from a stub sentence, every approval is denied at once without asking anyone, and agents a run delegates to are shadow runs too. A harness agent's calls are decided at the bridge between its session and the job: a call that does not only read answers from a stub. Its skill loads (the Claude Agent SDK's `Skill` tool) only read its session's read-only skills folder, so in a comparison they run on both arms, and a withheld skill is not in the arm's folder. A harness that cannot ask before its tool calls cannot run a shadow run: its runs are skipped and counted as skipped. ## Start one ```bash oz0 compare start drafter --variant draft=desk --nodes pool=dev --budget 1 --dry-run oz0 compare start drafter --variant draft=desk --nodes pool=dev --budget 1 --wait oz0 compare start keeper --variant withhold-skill=secret-word --budget 1 --wait oz0 compare start lead --variant inline-delegate=researcher --budget 1 --wait oz0 compare show --runs oz0 compare list ``` `--runs` sets the runs per arm and case (3 by default, at most 10), `--margin` the margin in points of pass rate (5 by default), `--confidence` the intervals' confidence (0.95 by default). A comparison spends the tenant's money and needs `--budget`; `--dry-run` shows the plan first and starts nothing. In the console: - **Studio:** with a draft of an installed plugin open, **More › Compare with the installed version**. - **Improve › Compare › Drafts and revisions** lists every comparison of a changed agent with its verdict, and starts one of another revision or a withheld skill. ## The verdict For each arm, the runs, the passes and the pass rate with its Wilson interval; then the change in pass rate, variant minus baseline, paired over the cases both arms ran: the cases are resampled with replacement 2,000 times from a seed of the comparison's own, so the result can be computed again, and the interval is widened to Newcombe's interval of the two pass rates where that is wider. - **Better:** the whole interval of the change lies above zero. - **Worse:** the whole interval lies below zero. - **Same:** the interval's lower end is above minus the margin, so the change is not worse than the installed version by more than the margin, and it is not shown better. - **Undecided:** fewer than three cases ran on both arms, or too few runs to tell; the result says how many runs would. The result lists the cases that **flipped**: those whose runs mostly passed on one arm and mostly failed on the other, each with both arms' passes, which is what a change won or lost case by case. Where an outcome check's judge named the rubric's criterion a failed answer missed, the result also counts each criterion's misses per arm, with the paired change in the share of runs that missed it, so you see which part of the answer the change made better or worse. Failures no judge explained are counted apart. The [evidence bundle](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#evidence-bundles) carries the tenant's finished comparisons of a changed agent as `comparisons-quality.jsonl`, in their own format (`oz0.comparison.quality/v1`: the method, the verdict, the criteria and every run, with no cost figure), apart from the measured savings in `comparisons.jsonl`. A comparison proves nothing about other cases, other times or another tenant's work, and nothing about what a stubbed write would have returned. ## A candidate check An agent-made changeset to an installed plugin is compared before anyone approves it, when the job's plugin holds a results grant for that plugin: the platform places it as a temporary private candidate (revision `check:`) and stores one comparison per changed agent, at most three, the installed version against the candidate, on the installed version's first 20 eval cases, one run per arm and case, graded by each case's `expect`. Each may spend at most $1, and no more than the job's budget has left. The runs are side-effect free, as every comparison's, and carry the job's taint. The verdicts, pass rates and intervals go beside the changes approval and into the changeset's record as `comparison:`; a comparison here is evidence for the person who decides, never an approval. See [the Plugin improver's candidate check](https://docs.oz0.ai/build/plugins/verified/plugin-improver/job-lifecycle#the-candidate-check). ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.