Understand runtime shutdown

For: cluster operators.

Review drain ordering, lost-session recovery and standard-stream compatibility.

The runtime's shutdown order

A node that stops drains first (Stops drain first): it takes no new work and waits for what runs there to end, within its stop grace of ten minutes by default. Then each runtime stops (an upgrade's old version once its jobs drained, a node restart, oz0-node stopping), in this order, within the 30 seconds of that grace the node gives it before it kills it:

1. It makes no new tool call: one that comes now waits, heartbeating, until the workers stop, and then fails as not made (ToolUnavailable); the job makes it again, and that attempt waits in the node's queue for the next runtime, however long that one takes to start.
2. It lets the calls it is stopping after a cancel report that they stopped, for at most 18 seconds (the 15-second heartbeat timeout and a margin), while its workers still take their jobs' workflow tasks, so those jobs see their calls stop; when there were such calls, it waits 1 second more for those jobs' workflow tasks.
3. Only then it stops its workers, which evicts the jobs they cache.

A runtime whose worker cannot finish evicting a job for 20 seconds logs it and exits with status 75, and the node starts a new one; the job goes on from its history there. temporalio 1.33 and 1.34 never finish such an eviction on their own when the job waits for a cancelled call (sdk-python#1938); the runtime carries the fix the issue proposes for those versions, so this exit is a backstop.

A plugin's tool server that does not start is started again, up to four times over about seven seconds, before the runtime takes work; meanwhile the node's status, oz0 dev and the Plugins and Nodes pages say which tool is not ready and why. A call of a tool whose server never started fails as not made (ToolUnavailable), which the job makes again up to four times over about 30 seconds, never as ToolNotFound, which it does not retry.

Lost harness sessions start again, and session calls' idempotency keys

From the release that starts lost harness sessions again, a node lost under a session no longer fails its job:

• A lost session starts again from scratch, or waits for a person. When a session's heartbeat stops for 30 seconds, or its node's runtime stops under it, the job starts a fresh session by itself when the agent is restart-safe, and otherwise waits for an approval of kind restart; a no or the deadline fails the job as a loss did before.
  The job takes a new home first. Scripts and alerts that expected such a job to fail now see it go on, or wait; oz0 plugin lint prints each harness agent's restart: verdict, and restart: ask in an agent's definition makes every loss wait for a person.
  A runtime that stops under a session now fails it as HarnessNodeStopped rather than as a cancel. Jobs that started on an earlier runtime keep failing at a loss, as their histories say.
• A harness session's calls of idempotent functions get new idempotency keys. A session's call of a function whose listing says idempotentHint now carries <job>/s<n>/<function>#<digest>/<k> (its place in the job) instead of <job>/<activity>, so a fresh session's repeat of a lost session's call has the lost call's key (the idempotency key). A server that keeps keys sees new keys for calls made after the upgrade; calls outside harness sessions, and of functions without the hint, keep <job>/<activity>. Check that functions you marked idempotent really are: a server that honours the key answers a fresh session's repeat from its record.
• A resumed remote session asks before it runs a call again. A remote agent whose session resumed after a lost node, and found a built-in call that was allowed but has no result, runs it again by itself only when it only reads; any other waits for a restart approval.
• Sessions that finish during a drain or stop are counted as drained in oz0_harness_sessions_total, and losses per node lower its health and show in oz0 doctor.

A session's streams are sockets

On Linux the runtime gives a harness, an ACP agent and a remote session's adapter and executor socket pairs for their standard streams instead of pipes, so that a session's tools cannot reach them (threat model).

• Install harness-claude 0.1.5 with the runtime. An older one runs, and keeps pipes to its CLI, which its tools can reach.
• A harness of your own works unchanged unless it opens /dev/stdin or /dev/stdout by name, which fails on a socket. Use the descriptors.
• Install remote-claude-managed 0.1.1 with the runtime. An older one keeps pipes to its shell, which that shell's processes can reach. In a remote session on Linux, a bash command that opens /dev/stdout or /dev/stderr by name, as echo x > /dev/stderr or tee /dev/stderr do, now fails with No such device or address. Use >&1 and >&2.
• A Python harness imports only from its own environment. The runtime starts one with -P and -s, so a harness that imported from its working directory no longer finds what it had there, and one on a Python before 3.11 does not start. The same goes for a session's own MCP server and a job's server that starts in its work folder. Install the package into the plugin's environment, and move to Python 3.11 or later.
• A session's /proc is read-only. A program in a session that writes to /proc/self/..., such as oom_score_adj, gets an error. Sessions with sandbox: {ptrace: true} keep a writable /proc.
• sandbox: {ptrace: true} needs kernel.yama.ptrace_scope of 1 or more on the node: elsewhere such a session is refused.