Check model changes and retirement

For: tenant administrators and cost owners.

Use canaries to detect changed provider behaviour and prepare for model retirement.

Model canaries

A pinned model name is not a promise that the model behind it stays the same: providers redeploy, quantize and reroute. A model canary notices when the answers behind a pinned name change, on one provider record of your tenant:

oz0 model canary on anthropic claude-sonnet-5-5
oz0 model canary list

PROVIDER   MODEL              STATE     BORDER INPUTS  LAST CHECK              NEXT              CALLS  COST     REASON
anthropic  claude-sonnet-5-5  watching  5              0.167 of 0.433          2026-10-10 03:12  294    $0.0039

1. Once: it sends 48 one-word prompts (digits, letters, common words, the same for every tenant and never your data) three times each, asking for one token at temperature 0, and keeps the first five whose answers split: a model's top answers are tied there. The method comes from a 2026 paper (arXiv 2602.11083), which reports that a change in the model shifts how such ties split; it is the paper's claim, measured on the models it studied. It then takes 30 answers of each as their reference. At most 294 one-token calls.
2. Each day: three answers to each of the five (15 one-token calls). Their distance from the reference, averaged, is held against what the reference itself gives 99 times in 100.
3. A finding when two days in a row differ: the model behind claude-sonnet-5-5 on anthropic changed: its answers at 5 border inputs differ since …; the provider reports claude-sonnet-5-5-20261009 (was claude-sonnet-5-5).
   It runs, at once, every eval routine of the tenant whose plugin has an agent that names the model, as provider/model, as a fallback or through an alias that maps to it; those runs say whether quality moved.
   The finding shows on the agents' pages, in oz0 eval findings and in oz0 doctor, and stays open until you reset the canary on the model as it is now: oz0 model canary reset anthropic claude-sonnet-5-5 --reason "...".

Good to know:

• Billed to your tenant. The calls go through the gateway like your agents': your key, your data rules and your tenant's budget, metered under the agent name oz0.canary. Only a tenant admin turns a canary on, and nothing is called until one does.
• What it can watch. Providers that speak Messages, chat completions or the Responses API (it uses the record's first; the Responses API takes at least 16 tokens, so the canary reads the first word). A model that refuses temperature 0, or reasons before it answers and hides it, cannot be watched this way: the canary says not supported with the reason. A route whose answers never split has no border inputs: a change there cannot be seen, and the canary says so.
• What it means. It sees that the answers at a few prompts changed, which includes changes to how a provider serves the model, not only the model. It does not say whether the model got better or worse: the evals do.
• What we have measured. Our tests check it against a stand-in model whose answers switch behind its name: the switch is found after two periods, and an unchanged stand-in raises nothing. How often it catches a real provider's change, and how often it raises a finding that is not one, we have not measured: treat a finding as a reason to look, and a quiet canary as no proof.
• --eval-canary-period on the server changes how often canaries check (a day by default).
• At most 50 per tenant. One driver checks every tenant's canaries, each tenant's most overdue first and then each one's next, so no tenant's canaries wait behind another's; turning on a 51st is refused until you turn one off.
• A change it finds also opens a drift finding with the cause model for each agent that uses the model, so each agent's page says what reached it.

When a model retires

Providers retire models on dates they announce, and every job of an agent that pins a retired model fails from that day. A provider record's model can say when, and what the provider names in its place:

oz0 provider model anthropic claude-sonnet-5-5 --retires 2026-12-31 --successor claude-sonnet-6
oz0 model list --retiring

PROVIDER   MODEL              CONTEXT  MAX OUTPUT  RETIRES                    SUCCESSOR
anthropic  claude-sonnet-5-5  1000000  64000       2026-12-31 (in 79 days)    claude-sonnet-6

Releases also ship the dates they know, as data in the release; a record's own date wins, and --retires "" clears it. Nothing is fetched to learn a date.

From 30 days before the date (oz0 tenant quality <tenant> --retirement-notice-days N, 1 to 365), each agent that pins the model gets a model_retiring finding: through its model, a fallback, a cascade step, an alias the tenant maps to the model, its outcome judge or its sampled judge. writer pins anthropic/claude-sonnet-5-5 through smart, which retires on 2026-12-31 (in 30 days); its successor is anthropic/claude-sonnet-6.

People are told with the model.retiring notification when the notice starts, at seven days and one day left, and on the date; oz0 doctor lists the open ones. The finding resolves itself once the agent pins another model (change its model, or point the alias elsewhere with oz0 model alias set), or the date is cleared.