Kill switches

For: tenant admins and governance reviewers.

Set a kill switch when an agent misbehaves, a tool leaks or a tenant's work must stop. The switch reaches every edge and node within seconds. The EU AI Act requires people who can stop high-risk systems; the control is also available for every other system.

oz0 kill agent acme-triage --reason "answers customers in a loop"
oz0 kill tool acme_github --reason "token leaked"
oz0 kill tool acme_github.merge --reason "merges without review"
oz0 kill tenant --tenant acme --reason "contract ended"

What stops

Agent and flow switches

No new jobs start (run, the runtime API, webhooks, quality gates, fires of routines): they are refused with the switch's reason, and the schedules of routines that run it pause until the switch is released.

The agent's model calls are refused at the gateway (403 oz0_killed; a park switch holds them with 503 oz0_parked instead), its calls of local models included, and its tool calls on the nodes, in every running job.

Its running jobs are parked or cancelled, child jobs included, and so is the agent's work inside another agent's job tree (a child another agent delegated to it).

A webhook that would start a stopped agent's or flow's job, a flow's or a routine's, answers 503 Service Unavailable with the switch and its reason, in the shape the gateway refuses model calls with:

{"type": "error", "error": {"type": "oz0_killed", "message": "kill switch kill-01a1000c-299b-778e-ac0b-fceac0eb3015 stops agent acme-triage: answers customers in a loop"}}

A sender that retries on 503 delivers again once the switch is released; a retried delivery with the same ID is still one job.

Every edge reloads the switches every second, and tells its nodes about tool switches on their control stream as soon as one changes. In our end-to-end test, new jobs were refused within 1.4 seconds of the switch, the running job was parked within 1.7 seconds, and the node refused the tool within 1.5 seconds.

A node never calls a tool while it cannot tell what the switches stop. It refuses every tool call until its supervisor has written the list the edge sent, and whenever the file cannot be read or the disk refused the write.

A node that just joined or started again must receive that list. A starting node removes the policy files of its last run, so a switch set while it was down never waits behind them.

The refusal says the kill-switch list has not reached this node yet, or that the list in its file cannot be read or is not what the edge sends. The node's log says why.

A supervisor whose write fails removes the old list rather than leave it in place, and writes the new one as soon as it can.

An edge that has not read the switches since it started sends its nodes no list, so a node keeps the one it has instead of one that releases what the edge cannot see.

Park or cancel

--mode says what happens to an agent's running jobs:

Park running jobs

park, the default, pauses them with Temporal's workflow pause, and pauses each of their activities too (a model call, a harness session, a step that waits for its retry): nothing more runs, and nothing is lost.

A model call that was already under way finishes, or fails and is made again after the release; a harness session that was running stops and runs again from its start after the release.

When the switch is released, the jobs it paused resume where they were, waiting approvals included, and their activities start again from their first attempt: the time parked uses up none of their retries.

A tool call that is already running when the switch comes is not paused by the workflow pause: the node checks the switches every second while a call runs, and cancels a call whose tool, agent or tenant the switch stops at its tool server.

The call then answers The call to acme_github.merge was stopped: a kill switch stops agent acme-triage, ..., as a tool's error: when the switch is released and the job resumes, the agent reads it and goes on.

A call that waited for its server checks the switches again before it is sent, and answers the same way.

So does a harness session's call of its plugin's own MCP server that waited for a person's decision: a switch thrown while the person decided stops it once the answer comes.

Cancel running jobs

cancel cancels them, as job cancel does.

oz0 kill agent acme-triage --mode cancel --reason "wrong prompt in production"

Setting --mode cancel on an agent that is parked already turns that switch into a cancel switch: its parked jobs are cancelled, without a moment where they resume. A cancel switch cannot go back to park.

Required Temporal support

Pausing needs Temporal's workflow pause and activity pause, which the embedded Temporal has on. With an external Temporal cluster, set frontend.WorkflowPauseEnabled in its dynamic config, and run a version with the activity pause API (PauseActivity); without them, use --mode cancel.

The fence: every call of a stopped job is refused at every edge

A switch is also a revocation. Each tenant has an authority epoch, a counter that every job's authority envelope records when the job starts and that its child jobs copy. A switch moves the tenant to the next epoch, and every edge refuses each call of a job the switch reaches whose envelope is older. New jobs get the new epoch only once its revocations have been computed, so a job that starts while a batch is still open is reached by every revocation that joins it. A cancel switch refuses:

• a model call through the gateway, a node's local admit, a child job's admission and a harness session's call through the node's proxy answer 403 with oz0_revoked;
• a connection's per-call token is refused;
• the job's workflow may not start a child job, schedule a tool call or open a session.

The edge finds the job by the call's binding, which it stamped itself, never by the Oz0-Agent header a node sends: a node that names another agent is refused all the same.

Model calls already running when an edge loads the switch are cut, and a streamed answer then ends with an error event of the same oz0_revoked error, so the call fails rather than giving the agent part of an answer as the whole of it.

A call cut while the gateway reads the tenant's provider key answers 503 with oz0_call_ended and the revocation's reason, never as a missing key: its activity tries again, the fence refuses that attempt, and the switch stops the job.

Every edge reloads the epochs each second; in our tests the edges fenced a revocation 0.2 to 0.8 seconds after it, on both edges of a split cluster.

{"type": "error", "error": {"type": "oz0_revoked", "message": "the job's authority was revoked at epoch 4 (agent acme-triage, revocation rev-01a1…): its calls are refused at the edge"}}

A park switch holds those calls instead of refusing them, so that its jobs pause and resume rather than fail: the gateway (also for a call that names the parked agent in Oz0-Agent) and the token route answer 503 with oz0_parked and Retry-After, and so do the routes a job's own steps call through the edge (filing an update, starting a candidate check, its evidence, provenance, change budget and mail, its files between tools), the workflow's next child start, tool call or session waits for the pause, and model calls already running finish.

The 503 only covers the moment between an edge loading the switch and the job's pause: the activity that met it is then paused with the job, so its retries wait for the release.

In our test, a job whose model calls met the park was held for 30 seconds, six times its Retry-After, and made its calls again 7 seconds after the release, then completed.

{"type": "error", "error": {"type": "oz0_parked", "message": "kill switch kill-01a1000c-299b-778e-ac0b-fceac0eb3015 stops agent acme-triage: answers customers in a loop"}}

The fence reaches exactly the jobs the switch stops. Jobs that started under a later epoch, and jobs the switch does not reach, are not refused, however old their envelope. Releasing a park switch lifts its fence, so the parked jobs resume and call again.

While the database does not answer

The fence fails closed. When an edge cannot read a job's envelope (the database or management is slow, busy or unreachable, a stored block fails its check, or the job's start is still being recorded), it holds what the fence checks instead of letting it through unchecked: the gateway answers 503 with oz0_envelope_unavailable and Retry-After, the token route answers 503, and the job's next child start, tool call, session or continue-as-new waits (its workflow task is offered again).

Jobs whose envelope is being recorded

A job whose start names an envelope the edge cannot read yet, or whose envelope was minted in the last ten minutes and is not recorded as started yet, is one with an envelope, and waits the same way.

An edge stores a job's envelope, and the record that the job started on it, before the start goes on, so every other edge reads it.

With the roles apart, an edge that finds no envelope of a job asks Temporal what the job's run names before it takes the job for one without: a job whose envelope is still on its way from the edge that started it (while that edge could not reach management) waits on the other edges until it arrives, and a scheduled fire one of whose tasks ran is never given a second envelope.

A job that started before envelopes existed has none to read and is not held, and nor is any job of a tenant with no live revocation, since nothing could reach it.

Management outages with separate roles

With the roles apart, no revocation can be recorded while either of these conditions holds:

• The edge gets no answer from management at all: its connection fails or nothing answers within 35 seconds. This lasts until management answers again, even if the answer says it is busy.
• Management cannot connect to its database at all. A slow or busy management or database, or one lost connection, does not count.

During that outage, the fence lets a call continue when the edge does not hold the job's envelope, then checks the next call again. A child start, a continue-as-new, a start that continues another job's authority and a call of another plugin's tool for such a job wait for management: they would change what the job may do for good.

In a tenant with models by level, a model call whose envelope's labels cannot be read waits too, management away included.

Once the database answers, the held calls go on, or are refused if a revocation reaches them.

New starts during an outage

A start whose envelope cannot be minted or stored is refused with Unavailable (try it again), and one whose agent's grants reach more tools and agents than one envelope holds (2,000) is refused with what to narrow.

A routine's scheduled fire that cannot get its envelope waits the same way (its first task is offered again), and one that never can (its routine has no such version, or its envelope would hold too much) is stopped and audited as routine.fire.refused with why; no job runs without an envelope for want of one.

Reports during an outage

What a node reports for a job (its usage reports, local models' calls, a secret's refusal, an outside agent's exchange, a harness's permission question) is written with the job's agent from its envelope, never the agent the node names.

While the edge cannot read the envelope it writes those reports without an agent rather than refuse them, since a node sends a report again only a few times and a refused one would be lost with its spend and its audit entry; it logs each one at warning level, naming the job.

A job that runs on no envelope reports with none.

Edges log each hold at warning level.

Exactly the jobs that lost their authority

Kill switches are one kind of revocation. The others stop jobs the same way, always by cancelling them, since the authority does not come back:

Retiring an agent in the inventory is not a revocation: it stops new use and lets running jobs finish. A tool's switch stops no job: the nodes refuse the tool's calls.

The edge that computes a revocation reads every running workflow of the tenant, of every type (agents' and flows' jobs, tool calls, channel posts, event deliveries), with no cap, and the envelope chain of each, and stops the jobs that match and no other.

Revocations that land within a moment of each other are one batch: one epoch step and one computation.

A job started before envelopes existed is matched by its agent for a kill, and listed as unknown for the other revocations.

A job whose envelope the edge cannot read is never taken for one of those: it is matched by its agent for a kill where that reaches it, and otherwise read again a few times; one still unreadable is on the receipt as indeterminate with why, the fence holds its calls meanwhile, and the sweep reads it again for ten minutes and stops it once it can tell.

A kill switch's pass reports such a job as an enforcement failure (oz0 kill list shows it) and leaves it parked until it can tell.

Temporal's list of running workflows shows a job a moment after it starts (seconds on a loaded cluster), so the edge also reads the edges' own record of the jobs they started from 30 seconds before the revocation on, and asks Temporal about each one the list does not show yet: a job started just before the revocation is stopped by its batch and on its receipt, whether it still runs or another path stopped it first.

A job started earlier than that window and still absent from Temporal's visibility can be missing from the receipt. Check visibility health before treating a receipt as a complete list of affected jobs.

Receipts

Each batch ends in a receipt: what it stopped, what the edges refused, and what could not be proven stopped. Management signs it with the audit key once the stop completes, usually within seconds. oz0 kill, oz0 plugin revoke and oz0 apikey revoke print the receipt's ID:

oz0 receipt list
oz0 receipt show rcpt-01a12280-…
oz0 receipt show rcpt-01a12280-… --out receipt.json
oz0 receipt verify receipt.json --key <fingerprint from oz0 audit key>

Receipt rcpt-01a12280-… (kill) of tenant default, epoch 2: signed
  revoked     agent helper (park)
  signed      2026-10-10 06:35 by audit key audit-3c1f… (3c1f…)
  computed    by edge-a: 9 running workflows read, 8 keep their authority
  fenced      edge-a 417 ms after the revocation (0 calls in flight cut)
  carriers    1 closed, 2 parked, 1 refused at fence, 1 indeterminate
  CARRIER         ID           AGENT   STATE                WHY
  approvals       -            -       parked               the waiting approvals and questions of the 1 parked jobs wait with them …
  job             job-9eae…    helper  parked (parked)      -
  model_calls     edge-a       -       refused_at_fence     the edge fenced the batch 417 ms after the revocation …
  routine         nightly      helper  parked               its schedules pause while the switch holds, and its fires are refused
  starts          -            -       refused_at_fence     StartJob, webhooks, event sources and routes, …

Each carrier ends in one of four states:

oz0 receipt verify checks the ed25519 signature over the receipt's document (every field, in its order) without the cluster or the network; pin the audit key's fingerprint you got from oz0 audit key by another channel.

The receipts whose audit entries fall in a period are also in the evidence bundle, whose verifier checks each one.

Management signs only what it has checked: the revocations, their first time, the edge that computed the batch, its completion, every fence, the counts and the kept jobs must be the batch's as management holds it, and the document must be in its canonical form (each field once, under its own name, written the way the edge's encoder writes it, and no field the format does not have), so that every reader, in any language, sees the values that were checked.

If management finds that the document an edge wrote is not a receipt of its own batch, it does not sign it: the receipt is rejected, with why, and oz0 doctor names it; its jobs were stopped all the same.

Keep the clocks of management and the edges within five minutes of each other (NTP): a receipt that an edge whose clock runs further ahead completes waits, unsigned, until management's clock catches up, and one dated more than an hour ahead is rejected, since its edge's clock is wrong.

Reading receipts requires audit.read (they name people and jobs); the console lists them under Govern, Kill switches, Receipts.

A receipt records what the platform did. Whether that meets a law or a contract's duty to be able to stop a system is for you and your advisers to judge.

See and release

oz0 kill list
oz0 kill list --all          # released ones too
oz0 kill release kill-01a1000c-299b-778e-ac0b-fceac0eb3015 --reason "fixed in 1.4.2"

ID                                         TENANT   STOPS         MODE  SINCE             BY              JOBS  REASON
kill-01a1000c-299b-778e-ac0b-fceac0eb3015  default  agent helper  park  2026-10-03 06:35  operator:admin  1     answers customers in a loop

JOBS is how many running jobs the switch paused or cancelled. Setting a switch that already holds returns that one. When an edge cannot pause or cancel some of the jobs (an external Temporal without workflow pause, say), the switch says so, in kill list and on the console, until the edge succeeds:

WARNING: kill-01a1000c-299b-778e-ac0b-fceac0eb3015 does not hold for every running job yet (2026-10-03 06:35): park job job-9eae…: …

The reason is at most 500 characters; keep the rest in your incident notes.

Setting and releasing take a reason (--reason, and optionally --decision-ref), which the entry keeps. Setting a switch applies at once; in a tenant with four eyes on, releasing one waits for another admin, and the switch holds until then. Setting, enforcing and releasing are in the audit trail as kill.set, kill.mode, kill.enforced (with the jobs), kill.release and kill.resumed, and each batch's signed receipt as kill.receipt (revoke.receipt for the other revocations).

Setting and releasing need the tenant's admin role; readers can list them. A switch a cluster admin set, during an incident or for abuse, only a cluster admin can release or change: the tenant's admins see it marked (cluster).

The console has them under Govern › Kill switches: Active lists what is stopped and, for those who may change governance, stops and releases with a reason; Receipts has the signed receipt of each, for those who read the audit log. The inventory marks what is stopped.

Link a switch to an incident

A switch set because something went wrong belongs in the incident that records it, with the jobs it parked or cancelled:

oz0 incident link inc-01a1300e-53f3-7dbc-bf41-c7f700105376 kill kill-01a1000c-299b-778e-ac0b-fceac0eb3015 --reason "the switch that stopped it"

The incident keeps the switch's time and a summary (what it stops, its mode, who set and released it), and its timeline export carries the switch's audit entries (kill.set, kill.release and the rest) so they verify offline. A receipt of the kill links the same way (oz0 incident link <id> receipt <rcpt-…>).