Recover a harness session

For: plugin authors.

Understand node loss and restart-safe sessions.

When a node is lost

A harness session is one long step on one node. When that node is lost under it, the job starts a fresh session: from the job's input, in new folders, on a node that runs the agent, as the same job with the same budget and the same authority. It does not continue the lost conversation, and what the lost session spent stays spent. A job starts at most two fresh sessions; the next loss fails it.

Before the fresh session starts, the job takes a new home: the node where the session's calls of your tools, its skill reads, its waits for events, its changeset and, in a job that needs a tier-1 node, its hooks run. The lost node may have been the job's home, and nothing would run there until it came back.

The new home is a node that runs the agent (a tier-1 node for a job that needs one); the job keeps the tool policy it started with. The fresh session's calls do not count toward the lost session's loops: repeating them is what a fresh start does.

A session counts as lost when:

• its heartbeat stops for 30 seconds (the node crashed, froze or lost its network);
• the node's runtime stops under it (a drain's deadline ended, or a stop's grace ran out before the session finished).

Plan a node's maintenance with oz0 node drain NODE --deadline 2h first, so sessions finish rather than start again. A session that fails by itself, or one cancelled with its job, is no loss: the job fails or ends as before.

The fresh session starts by itself only when the agent is restart-safe (below). Otherwise a person decides: the job waits for an approval of kind restart that names the node, how far the lost session got, and the calls it was allowed whose effect nobody can see, by tool and a digest of their arguments (calls of its last seconds may be missing). Yes starts the fresh session; no, or no answer by the deadline, fails the job. A deadline never says yes.

oz0 job trace shows both sessions: the lost one as lost with its node, the fresh one as (fresh session 1); a later outcome round's session is a session of its own, not another restart. oz0 job why says what became of each lost session, and the audit log has run.session.lost and run.session.restarted. Each loss is also counted per node and agent.

A lost session that still runs somewhere (a node that only lost its network) cannot act through the job any more: its calls are refused once a fresh session runs.

A session that finishes while its node stops counts as drained when the edge received the last heartbeat or the node returns within ten minutes. If the node stays away longer, that completion is not counted as drained.

Restart-safety

An agent restarts by itself when a fresh session cannot repeat, unseen, something the lost one did outside. oz0 plugin lint proves it per agent and prints the verdict:

reader: restart: automatic
  when the node ran the session in a sandbox (tier 1), so its file changes stayed in its own folders
coder: restart: ask
  because Bash can reach api.example.com from the session's sandbox, and could change something there that a fresh session repeats

Every outside effect of a restart-safe agent goes through one of:

• a changeset (changes:), applied only after review; a lost session never returned one, so nothing of it is applied;
• a safe output of a routine, applied by the edge after the run (not one the routine applies during the fire);
• a read-only tool (readOnlyHint), or the harness's own tools that only read (Read, Glob, Grep);

an idempotent tool

The tool declares idempotentHint, and its server must honour the idempotency key. A session's call of such a tool carries its place in the job as its key: <job>/s<n>/<function>#<digest of its arguments>/<k>, the k-th call of that function with those arguments in the job's n-th session (r<m>/ comes first after the job continued as new). A fresh session after a loss keeps the lost session's n and counts k again from one, so its first label(bug) carries the lost session's first label(bug)'s key and does nothing new, while a second label(bug) after a label(fix) is a call of its own.

A later outcome round or cascade step is a session of its own, with a new n. See the idempotency key.

The Skill tool, which a Claude Agent SDK session of an agent with skills gets to load them, only reads the session's read-only skills folder: it counts as a read on any node. The harness's tools that change files (Write, Edit) or run commands (Bash) are restart-safe only inside the session's sandbox, where a change stays in the session's own folders, and a command reaches no host: the agent, its plugin, its harness's plugin and the tenant allow no egress for it.

The session must also hold no credential bound to its sandbox and no secret of its harness's plugin, and run no MCP server of the plugin inside it (an imported agent's servers, when its install allows them in sessions, are such servers; its hooks join the job's hooks and run for the fresh session's calls as for any). Credential bindings of the plugin's tools (requires.inject) reach only those tools, whose calls are checked as above. Questions to people, colleagues and the job's board may come again in a fresh session.

Lint sees only the plugin's files. What only a node knows (whether the session ran in a sandbox, the tenant's egress, credentials bound to it, another plugin's function listing) the job checks again when the session is lost, from what the session reported of itself, and asks a person when one of them fails.

To always ask, set restart: ask:

restart: ask      # every lost session waits for a person; nothing can make an agent restart by itself

A harness of your own: a session that its runtime stops reports why, and the platform knows which tools only read only for the harnesses it knows (the Claude Agent SDK, and ACP's tool kinds); any other harness's lost sessions ask.