Keep calls compatible during upgrades

For: cluster operators.

Check which requests older runtimes can still make during an upgrade.

Call bindings and nodes on an older runtime

From this release the edge stamps each activity it hands a node with a call binding, and refuses a model call, a child's admission or a connection token that names a job without one (run receipts). A node whose runtime predates bindings sends none.

By default (--call-binding auto) the edges notice: while a node heard from in the last ten minutes reports a runtime without call bindings, or a node binary too old to report what its runtime does, they let that node's calls through without one, for the jobs whose tasks the edge handed that node in the last 30 minutes, so jobs waiting across the upgrade go on; every other node's calls still need their binding.

Once a node runs this release's runtime and binary, they require bindings of it by themselves, within a few seconds, and log it once every node does.

From then on the node's later heartbeats no longer change that: only telling the node to run another runtime or binary does (oz0 node upgrade, a rollout or its rollback).

oz0 doctor names the nodes it waits for. Upgrade the nodes' binaries and runtimes after the servers (oz0 node upgrade <node> --binary, then --runtime <version>), then pin --call-binding required, which takes nothing on a node's word.

An explicit flag wins: --call-binding required refuses calls without a binding at once, whatever the nodes report (jobs that waited across the upgrade on an older runtime then fail at their next model call with 403 oz0_binding ... upgrade the node's runtime), and --call-binding when-present keeps taking them until you drop it.

A node's reports and permission asks carry their call's binding

From this release the edge takes the job of a node's usage report, local model report, audit report (an outside agent's exchange, a file write, a credential's first use, a call refused for a secret's value) and harness permission ask from the call binding the request carries, as it does a model call's, and refuses one that names another job.

A harness session's reserved tools (its quota wait, an ACP agent's call of a declared tool, a colleague's question) run only for an ask with the runtime's own copy of the binding, which the edge stamps beside the binding from this release on; the node's proxy no longer marks the runtime's asks with a header.

A node on an older runtime sends no binding with these requests: under --call-binding when-present, and under auto from a node that itself runs an older runtime, the edge takes them only from that node and only for the very job whose task it handed that node in the last 30 minutes, which the task's heartbeats keep (a report after a longer wait without one, for another job of the same tree, or after the edge restarted, is refused with 403 ... upgrade the node's runtime).

A harness session's permission ask without a binding is taken only from the node that got the session's latest task, under update IDs of its own, so an ask that was waiting across the upgrade is asked again once.

No ask of an older runtime is the runtime's own, so a harness session's reserved tools are refused there until the node runs this release's runtime.

A model call, a connection token or a live-stream publish without a binding follows the same rule: only for a job the edge handed that node a task of lately.

A credential use reported outside any job must name a tool the node runs.

Upgrade the nodes' runtimes after the servers, as above.

Job IDs are idempotency keys, and the admin API's Idempotency-Key

From this release a job ID a client chose is the start's idempotency key (job IDs and retries):

• An ended job never runs again under its ID. A start under the ID of a job that ended answers with that job when it repeats the request, and is refused with already_exists (oz0_id_conflict) when it does not. A script or app that reused an ID on purpose to run a job again must start each run under a new ID (or without one).
• Another request under a running job's ID is refused instead of answered with the running job; the same request again is answered with retried: true.
• Jobs started before the upgrade carry no record of their request, so nothing shows a start is the same one: a start under the ID of one is refused with already_exists, running or ended.
• Starts are compared per caller. The caller is part of the request, so a start under someone else's job ID is refused, where it used to join their running job.
• oz0 run, the SDKs and the console send an Idempotency-Key on every start without an ID and retry unavailable with it; the job's ID is then derived from the key, and looks the same as before (job- and a UUID).

The admin API takes an Idempotency-Key header on the calls that have no natural key (the admin API). Management keeps each key's request and answer for 24 hours in a new table, idempotency_keys (migration 221), pruned every hour by one server at a time; a backup leaves it out, and so does a tenant export. Answers that carry a credential are kept sealed with a key derived from the client's own Idempotency-Key. Expect a row per keyed call of the last day: the CLI and the console send a key on every such call.

Join tokens are bounded, and waiting nodes too

A join token now admits at most 1,000 nodes and lasts at most 30 days; one made without --max-uses or --ttl admits one node for 24 hours.

A token made before the upgrade with no use limit (--max-uses 0) or no expiry (--ttl 0), or with more than these bounds, admits nobody after it: the join it refuses names the token and the commands that replace it, and oz0 token list shows it as unbounded. Make a bounded one with oz0 token create and revoke the old one. Nodes that already joined with such a token keep working.

A script that passed --max-uses 0 or --ttl 0 to the CLI now fails with the reason; a client of the admin API that sends a max_uses of 0 gets a token of one use (the answer names it), and one that sends a ttl of 0 is refused.

Dev mode's token in join-token is made again at the next start (1,000 uses, 30 days).

Nodes that join without a token wait in a bounded list: at most 1,000 per region at once, at most 20 of them from one address, and a place lasts 24 hours from the first ask, however often the node asks again. A full list or a full share refuses further joins without a token, with the reason. Nodes waiting at the upgrade count against no address until they join again.