Improve

For: People testing and improving agents.

Start with the question you need to answer. Use the same cases when comparing a change, and keep failures as regression cases.

• Why did this job fail?: Find recurring failures and open the affected jobs.
• What changed?: Compare jobs and agent versions.
• Where should I fix it?: Choose the part that can correct the failure at the lowest recurring cost.
• Does the change pass?: Write eval cases and gate plugin updates.
• Is the answer good enough?: Configure judges, reviewers and candidate selection.
• Is the new definition better?: Compare quality on paired cases.
• Does it cost less?: Measure billed cost per successful run.
• How do the starters perform?: Run and inspect the starter agents' evals.
• Can an agent inspect the evidence?: Grant bounded read access to results.