--- title: "Audit trail" description: "What every run did and who changed what, in a log that cannot be edited unnoticed, kept as long as your rules say and exported as JSON lines." canonical_url: "https://docs.oz0.ai/govern/audit-trail" --- # Audit trail > What every run did and who changed what, in a log that cannot be edited unnoticed, kept as long as your rules say and exported as JSON lines. For: tenant admins and governance reviewers. Read the audit trail to identify who acted, what authority a run used and which decisions the platform recorded. Export the records for review, or use an evidence bundle when you need to verify a job's receipts and gateway records together. The audit log records what people change (tenants, nodes, plugins, secrets, members, rollouts), what the platform does by itself (gate verdicts, health changes), and what runs do. It is meant for the person who has to answer for what an agent did: an auditor, a regulator, a customer's security team. An authority's question becomes an export, not a project. ## Export ```bash [Terminal] oz0 audit export --tenant default -o audit.jsonl oz0 audit verify --file audit.jsonl --key 3f9a… ``` An export is one chain as JSON lines, oldest first. Each line carries the entry and its place in the chain, and the last line is the chain's head when the export was made: ```json [audit.jsonl] {"seq":7,"id":"run-4f0c…","at":"2026-10-03T06:34:12.101Z","at_ms":1790000052101,"recorded_at":"2026-10-03T06:34:12.240Z","recorded_at_ms":1790000052240,"tenant":"default","actor":"node:node-486fffd604bd","action":"run.model_call","target":"job-9eae…","agent":"helper","detail":{"model":"claude-sonnet-5-5","provider":"anthropic","status":200},"salt":"0f3a…","actor_hash":"77c2…","target_hash":"1d09…","detail_hash":"a4e8…","private_hash":"c61b…","prev_hash":"9c1e…","content_hash":"4b7d…","hash":"e02a…"} {"head":{"tenant":"default","seq":45,"hash":"9c41…","base_seq":0,"base_hash":"","seal":{"blinded":0,"lines_sha256":"7be0…","key_id":"audit-3f9a…","public_key":"mC4w…","signature":"Q2hr…"}}} ``` `audit verify --file` checks an export without the cluster, so an auditor can check what you handed over: every entry, that the entries end at the head line, that the first one links to the chain's base when the export starts there, and that every redacted entry is listed by the erasure or retention entry that redacted it. Lines cut from the end, or from the start of a whole export, break it. It prints the head; compare it with the one `audit verify` shows on the cluster, or with a copy you kept elsewhere. For an export with [blinded entries](https://docs.oz0.ai/govern/audit-trail/records#what-a-job-read-or-wrote) that comparison is not enough, so such a file fails until `--key` pins who signed it. `--complete` also fails unless the file holds the whole chain, from its base to its head. `--key` and `--complete` each fail unless every head in the file is signed and the signature verifies, and unless no entry comes after the last head: hashes alone show only that a file agrees with itself, since whoever edits one can work them out again, head included. With `--key` the signature must be that key's. Anyone can sign, so `--complete` without `--key` is refused unless `--unpinned` says that showing the file is whole is enough, as for a file with blinded entries. The output calls a file `complete` only when `--key` pinned its signature; a whole file whose key was not pinned reads `reaches from the chain's base to the head it names`, followed by a line that says the key was not pinned. A `--key` with nothing in it, as an unset variable in a script gives, is refused rather than read as no key, and so are `--key`, `--complete` and `--unpinned` without `--file`, and `--tenant` or `--cluster` with it: none of those could check what they say. The other commands that pin the audit key (`evidence verify`, `incident verify`, `receipt verify`, `tenant export --verify` and `subject export --verify`) refuse an empty `--key` too, and an export that is given a `--key` without `--verify` is refused. An export from a server that did not sign plain heads passes neither flag, and without them is checked as far as its hashes go: the command says `unsigned`, and does not call it complete. Export it again to get a signed head. ```text [Output] ok: entries 1 to 45 of chain default chain correctly (2 redacted, each listed by the erasure or retention entry that redacted it) complete: from the chain's base to its head, entry 45, hash 9c41… compare the head with `oz0 audit verify` on the cluster, or with a copy kept elsewhere signed by audit key 3f9a… the audit key is the one given with --key ``` To check an entry yourself: each part hash is the SHA-256 of the salt and the part, each written as its length in bytes, a colon and the bytes; the parts are `actor`, `target`, `detail` and `private` as the JSON text in the line (empty when there is none), on a version 2 or 3 entry `starter` too, and on a version 3 entry `granted_by`. The hash is the SHA-256 of `prev_hash`, a colon and `content_hash`. A line marked `withheld` left its private part out; its `private_hash` is checked as it is. A line marked `blinded` has no salt, so only its `content_hash` and links can be checked this way, and its export's signature covers the rest. On a line that also says `binding_keyed`, the call binding is keyed and the `content_hash` cannot be worked out again, so only its link is checked this way. ### A time range `oz0 audit` lists the newest entries first, in the chain's order: one tenant's by sequence number, several tenants' by the time the database recorded them, then tenant and sequence number. `--since` and `--until` keep those recorded in a time range (RFC 3339, or how long ago such as `2h`), as `audit export` does: ```bash [Terminal] oz0 audit --tenant default --since 2026-10-12T08:00:00Z --until 2026-10-12T10:00:00Z --action egress. oz0 audit --since 30m --target job-9eae… ``` The list never orders by an entry's `at`, which is when it happened by the clock of whoever wrote it: management, an edge or the job's history. Those clocks differ a little, an edge may hold entries for a while when management is away, and a clock can step back (NTP does that on servers, and WSL's clock steps back about a second every half minute), so two entries' `at` can disagree with the order they were recorded in. Each entry still shows its `at`. The recorded time comes from the database's one clock and never goes back along a chain, so a time range is a whole piece of it. A page token from before this order (an older server's) is refused with "ask for the first page again". An [incident](https://docs.oz0.ai/govern/incidents) links a range by time or by sequence numbers, and its timeline export carries the range as a whole piece of the chain, with a checkpoint signed by the audit key, so it verifies offline. ### Three versions of an entry Entries come in three versions, and a chain holds them all: those written before the starter was recorded are version 1, those written until the authority envelope was recorded are version 2, and those written since are version 3. Each keeps its version, and `audit verify`, on the cluster and with `--file`, checks each by its own rules, so an export that crosses a change verifies in one pass, without a flag.
Version In the export Its content hash covers, in this order
1 no "v" field seq , id , at_ms , recorded_at_ms , tenant , action , subject , agent , actor_hash , target_hash , detail_hash , private_hash
2 "v": 2 , and starter , starter_hash , agent_version , plugin and tool when set the text v2 , then the fields of version 1 up to agent , then agent_version , plugin , tool , then the four part hashes, then starter_hash
3 "v": 3 , the fields of version 2, and envelope , binding , level , granted_by , granted_by_hash , granted_at_ms , level_policy_version , action_class and labels when set the text v3 , then the fields of version 2 up to tool , then envelope , binding , level , granted_at_ms , level_policy_version , action_class , labels , then the four part hashes, then starter_hash , then granted_by_hash
The leading `v2` or `v3` keeps an entry from verifying under another version: stripping an entry of its newer fields, or giving an older entry a starter or a level, breaks the chain there. A version 1 or 2 line is exported exactly as before.
Flag What it does
--tenant The tenant's chain; without it, cluster admins export the cluster's own
--after N Only entries after sequence number N, to go on from the last export
--since , --until Only entries recorded in that time (RFC 3339): the piece of the chain from the first such entry to the last
--follow Keep running and write new entries as they are recorded
### To a SIEM `--follow` writes new entries as they come, every two seconds, each batch followed by a head line, so any log shipper that reads a command's output or a file sends them on: ```bash [Terminal] oz0 audit export --tenant default --follow >> /var/log/oz0/audit.jsonl ``` Every export is in the chain it exports (`audit.export`). A follow is recorded once, as `audit.follow` when it starts, not at every poll, so a SIEM that follows a chain does not fill it with its own reads. The console's audit page has **Verify** and **Export** buttons that do the same for the chain the page shows. ## Receipts of kills and revocations Each [kill switch and revocation](https://docs.oz0.ai/govern/kill-switches#receipts) ends in a receipt, which management signs with the audit key and records as an entry: `kill.receipt` (a batch that holds a kill switch), `revoke.receipt` (a grant narrowed, a plugin removed, an API key revoked, a person who can no longer start jobs) or `demotion.receipt` (reserved for the autonomy levels' circuit breaker; nothing writes one yet). The entry's target is the receipt's ID (`rcpt-`, which an incident can link) and its detail has the epoch, what was revoked, the carriers by final state, the key and the SHA-256 of the signed document. The edges' refusals of a revoked job's calls are entries too (`run.revoked_refused`), and an edge that stops a job a recent revocation reaches after the batch (one that started on an edge that had not loaded it yet) records `revoke.enforced`. `oz0 receipt show` and `verify` read them; an [evidence bundle](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#evidence-bundles) carries the receipts its segment's receipt entries name in `receipts/receipts.jsonl` (an entry is written when its receipt is signed, so a batch that opened before the period and was signed in it is there), and `oz0 evidence verify` checks each one's signature and that every receipt entry in the segment names a receipt the bundle holds, with the same document. ## Reasons and governance changes Every change of a governance setting (retention, legal holds, kill switches, the inventory's risk class, data rules, label settings and label policies, the AI label, the tool policy, the sandbox and egress settings, works-council mode, what notifications carry, CORS origins, four eyes, the erasure policy and agents' keep-for periods) carries the reason its admin gave and, when given, a decision reference: the entry's detail has `reason`, `decision_ref`, `setting` and `direction` (`tighten`, `loosen` or `same`). Entries written before reasons were required stay as they were. A change that waited for a second admin lands as the approver's entry with `change` and `requested_by`, after `governance.change_requested` and `governance.change_approved` (or `governance.change_rejected`) and before `governance.change_applied`, which hold the setting's value before and after, what loosens and the reason (for label policies, the people listed only as a count: the change request itself, which only admins read, keeps who) ([four eyes](https://docs.oz0.ai/govern/reviewed-changes#four-eyes-for-governance-changes)). `toolpolicy.would_have` records each call a [shadow rule](https://docs.oz0.ai/govern/tool-policy/review-changes#shadow-mode-and-ends) would have decided: the rule's ID, the agent, the tool, its action class, what the rule would have said and what happened. ## Related tasks and reference - [Audit records and run identity](https://docs.oz0.ai/govern/audit-trail/records) - [Verify the audit chain](https://docs.oz0.ai/govern/audit-trail/verification) - [Export and verify evidence bundles](https://docs.oz0.ai/govern/audit-trail/evidence-bundles) - [Audit retention, erasure and legal holds](https://docs.oz0.ai/govern/audit-trail/retention) a bundle made without the content permission : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#a-bundle-made-without-the-content-permission). a chain that shows edits : [Read the section](https://docs.oz0.ai/govern/audit-trail/verification#a-chain-that-shows-edits). checking a bundle : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#checking-a-bundle). erasures in the audit log : [Read the section](https://docs.oz0.ai/govern/audit-trail/retention#erasures-in-the-audit-log). evidence bundles : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#evidence-bundles). legal hold : [Read the section](https://docs.oz0.ai/govern/audit-trail/retention#legal-hold). legal holds and temporals histories : [Read the section](https://docs.oz0.ai/govern/audit-trail/retention#legal-holds-and-temporals-histories). on whose authority and at what level : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#on-whose-authority-and-at-what-level). retention : [Read the section](https://docs.oz0.ai/govern/audit-trail/retention#retention). retention holds and erasure : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#retention-holds-and-erasure). searching by starter : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#searching-by-starter). the audit key and the gateway keys : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#the-audit-key-and-the-gateway-keys). the gateways own record : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#the-gateways-own-record). the two way check : [Read the section](https://docs.oz0.ai/govern/audit-trail/evidence-bundles#the-two-way-check). what a job read or wrote : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#what-a-job-read-or-wrote). what a run leaves : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#what-a-run-leaves). when an entry cannot be written : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#when-an-entry-cannot-be-written). who started it : [Read the section](https://docs.oz0.ai/govern/audit-trail/records#who-started-it). ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.