---
title: "Keep calls compatible during upgrades"
description: "Check which requests older runtimes can still make during an upgrade."
canonical_url: "https://docs.oz0.ai/fleet/upgrades/call-compatibility"
---
# Keep calls compatible during upgrades

> Check which requests older runtimes can still make during an upgrade.

For: cluster operators.

Check which requests older runtimes can still make during an upgrade.

## Call bindings and nodes on an older runtime

From this release the edge stamps each activity it hands a node with a call binding, and refuses a model call, a child's admission or a connection token that names a job without one ([run receipts](https://docs.oz0.ai/concepts/run-receipts)). A node whose runtime predates bindings sends none.

By default (`--call-binding auto`) the edges notice: while a node heard from in the last ten minutes reports a runtime without call bindings, or a node binary too old to report what its runtime does, they let that node's calls through without one, for the jobs whose tasks the edge handed that node in the last 30 minutes, so jobs waiting across the upgrade go on; every other node's calls still need their binding.

Once a node runs this release's runtime and binary, they require bindings of it by themselves, within a few seconds, and log it once every node does.

From then on the node's later heartbeats no longer change that: only telling the node to run another runtime or binary does (`oz0 node upgrade`, a rollout or its rollback).

`oz0 doctor` names the nodes it waits for. Upgrade the nodes' binaries and runtimes after the servers (`oz0 node upgrade <node> --binary`, then `--runtime <version>`), then pin `--call-binding required`, which takes nothing on a node's word.

An explicit flag wins: `--call-binding required` refuses calls without a binding at once, whatever the nodes report (jobs that waited across the upgrade on an older runtime then fail at their next model call with `403 oz0_binding ... upgrade the node's runtime`), and `--call-binding when-present` keeps taking them until you drop it.

## A node's reports and permission asks carry their call's binding

From this release the edge takes the job of a node's usage report, local model report, audit report (an outside
agent's exchange, a file write, a credential's first use, a call refused for a secret's value) and harness permission
ask from the call binding the request carries, as it does a model call's, and refuses one that names another job.

A
harness session's reserved tools (its quota wait, an ACP agent's call of a declared tool, a colleague's question)
run only for an ask with the runtime's own copy of the binding, which the edge stamps beside the binding from this
release on; the node's proxy no longer marks the runtime's asks with a header.

A node on an older runtime sends no
binding with these requests: under `--call-binding when-present`, and under `auto` from a node that itself runs an
older runtime, the edge takes them only from that node and only for the very job whose task it handed that node in
the last 30 minutes, which the task's heartbeats keep (a report after a longer wait without one, for another job of the
same tree, or after the edge restarted, is refused with `403 ... upgrade the node's runtime`).

A harness session's
permission ask without a binding is taken only from the node that got the session's latest task, under update IDs of
its own, so an ask that was waiting across the upgrade is asked again once.

No ask of an older runtime is the
runtime's own, so a harness session's reserved tools are refused there until the node runs this release's runtime.

A model call, a connection token or a live-stream publish without a binding follows the same rule: only for a job the
edge handed that node a task of lately.

A credential use reported outside any job must name a tool the node runs.

Upgrade the nodes' runtimes after the servers, as above.

## Job IDs are idempotency keys, and the admin API's Idempotency-Key

From this release a job ID a client chose is the start's idempotency key
([job IDs and retries](https://docs.oz0.ai/build/apps/runtime-api#job-ids-and-retries)):

- An ended job never runs again under its ID. A start under the ID of a job that ended answers with that job when
it repeats the request, and is refused with `already_exists` (`oz0_id_conflict`) when it does not. A script or app
that reused an ID on purpose to run a job again must start each run under a new ID (or without one).
- Another request under a running job's ID is refused instead of answered with the running job; the same request
again is answered with `retried: true`.
- Jobs started before the upgrade carry no record of their request, so nothing shows a start is the same one: a
start under the ID of one is refused with `already_exists`, running or ended.
- Starts are compared per caller. The caller is part of the request, so a start under someone else's job ID is
refused, where it used to join their running job.
- `oz0 run`, the SDKs and the console send an `Idempotency-Key` on every start without an ID and retry
`unavailable` with it; the job's ID is then derived from the key, and looks the same as before (`job-` and a UUID).

The admin API takes an `Idempotency-Key` header on the calls that have no natural key
([the admin API](https://docs.oz0.ai/reference/admin-api)). Management keeps each key's request and answer for 24 hours in a new table,
`idempotency_keys` (migration 221), pruned every hour by one server at a time; a backup leaves it out, and so does a
tenant export. Answers that carry a credential are kept sealed with a key derived from the client's own
`Idempotency-Key`. Expect a row per keyed call of the last day: the CLI and the console send a key on every such call.

## Join tokens are bounded, and waiting nodes too

A join token now admits at most 1,000 nodes and lasts at most 30 days; one made without `--max-uses` or `--ttl` admits one node for 24 hours.

A token made before the upgrade with no use limit (`--max-uses 0`) or no expiry (`--ttl 0`), or with more than these bounds, admits nobody after it: the join it refuses names the token and the commands that replace it, and `oz0 token list` shows it as unbounded. Make a bounded one with `oz0 token create` and revoke the old one. Nodes that already joined with such a token keep working.

A script that passed `--max-uses 0` or `--ttl 0` to the CLI now fails with the reason; a client of the admin API that sends a `max_uses` of `0` gets a token of one use (the answer names it), and one that sends a `ttl` of `0` is refused.

Dev mode's token in `join-token` is made again at the next start (1,000 uses, 30 days).

Nodes that join without a token wait in a bounded list: at most 1,000 per region at once, at most 20 of them from one address, and a place lasts 24 hours from the first ask, however often the node asks again. A full list or a full share refuses further joins without a token, with the reason. Nodes waiting at the upgrade count against no address until they join again.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.