---
title: "Export and verify evidence bundles"
description: "Export and verify evidence bundles for tenant admins and governance reviewers."
canonical_url: "https://docs.oz0.ai/govern/audit-trail/evidence-bundles"
---
# Export and verify evidence bundles

> Export and verify evidence bundles for tenant admins and governance reviewers.

For: tenant admins and governance reviewers.

## Evidence bundles

An evidence bundle is what you hand an auditor for a period: files they check on their own machine, without the cluster and without the network.

```bash [Terminal]
oz0 audit key                                          # the audit key's fingerprint, to give the auditor by another channel
oz0 evidence export --tenant acme --from 2026-07-01T00:00:00Z --to 2026-10-01T00:00:00Z --out acme-q3.zip
oz0 evidence verify acme-q3.zip --key 3f9c…            # on the auditor's machine
```

<table>
<thead>
  <tr>
    <th>
      In the bundle
    </th>
    
    <th>
      What it is
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        audit/segment.jsonl
      </code>
    </td>
    
    <td>
      The tenant's audit chain for the period, entries of <a href="https://docs.oz0.ai/govern/audit-trail#three-versions-of-an-entry">
        every version
      </a>
      
      ; private parts, digests as stored and salts only when you have the content permission (<a href="https://docs.oz0.ai/govern/audit-trail/evidence-bundles#a-bundle-made-without-the-content-permission">
        without it
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        envelopes/blocks.jsonl
      </code>
    </td>
    
    <td>
      Every block of every <a href="https://docs.oz0.ai/concepts/run-receipts">
        authority envelope
      </a>
      
       the segment's entries name, with their parents back to block 0, one a line with its hash
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        audit/checkpoint.json
      </code>
    </td>
    
    <td>
      Where the segment sits in the chain (the entry before it, its last entry, and the chain's head when it was exported), and the same for the gateway's chain, signed with the audit key
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        gateway/records.jsonl
      </code>
    </td>
    
    <td>
      The <a href="https://docs.oz0.ai/govern/audit-trail/evidence-bundles#the-gateways-own-record">
        gateway's own record
      </a>
      
       of every model call of the period, each signed
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        gateway/jobs.json
      </code>
    </td>
    
    <td>
      The jobs those records name that started before the period, as the cluster knows them
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        node/calls.jsonl
      </code>
    </td>
    
    <td>
      Each ended job's <code>
        run.calls
      </code>
      
       entry, one job a line
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        reconciliation.json
      </code>
    </td>
    
    <td>
      The <a href="https://docs.oz0.ai/govern/audit-trail/evidence-bundles#the-two-way-check">
        two-way check
      </a>
      
       as the cluster computed it
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inventory.json
      </code>
    </td>
    
    <td>
      The <a href="https://docs.oz0.ai/govern/inventory">
        inventory
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        plugins/&lt;name>@&lt;version>.json
      </code>
      
      , <code>
        .cdx.json
      </code>
    </td>
    
    <td>
      A factsheet and available SBOM for each plugin version installed or used during the period. See <a href="#plugin-evidence">
        plugin evidence
      </a>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        models/&lt;provider>__&lt;model>.json
      </code>
    </td>
    
    <td>
      A factsheet per model the tenant's agents may run on and per model it vetted, as at the export: what it is reached through, its publisher, source, endpoint and base, a local model's digest and whether its nodes confirmed it against a pin, what that rests on, and its <a href="https://docs.oz0.ai/spend/models-and-keys/data-rules#provenance-and-vetting">
        vetting
      </a>
      
       (who, when, why). A vetting's own entry (<code>
        provider.model_vetted
      </code>
      
      , <code>
        provider.model_unvetted
      </code>
      
      ) is in <code>
        governance.jsonl
      </code>
      
      . A vetting is a person's judgement, not a scan of the weights, and the factsheet says so
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        policy.json
      </code>
      
      , <code>
        budgets.json
      </code>
    </td>
    
    <td>
      The tool policy and the budgets in force at the export
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        retention.json
      </code>
    </td>
    
    <td>
      Retention, legal holds, what Temporal keeps of histories and whether payloads are encrypted
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        approvals.jsonl
      </code>
      
      , <code>
        governance.jsonl
      </code>
    </td>
    
    <td>
      The period's approvals and governance changes (tool policy, kill switches, retention, members, plugins, with their reasons where they carry one), copied from the segment
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        keys/
      </code>
    </td>
    
    <td>
      The public keys that signed the manifest, the checkpoint and the gateway's records
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        manifest.json
      </code>
      
      , <code>
        manifest.sig
      </code>
    </td>
    
    <td>
      Every file's size and SHA-256, signed with the audit key
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        README.txt
      </code>
    </td>
    
    <td>
      What the bundle holds and how to check it
    </td>
  </tr>
</tbody>
</table>

### Plugin evidence

Each plugin version installed or used in the period gets a factsheet (its source and commit or digest, what was checked of its signature, its [version state](https://docs.oz0.ai/build/plugins/indexes/version-records#version-states), its gate and its [scheduled eval runs](https://docs.oz0.ai/improve/evals/regressions#scheduled-evals) (counts, cost, models and verdict, never a case), the tool policy rules that name its tools and the grants that involve it), and its [SBOM](https://docs.oz0.ai/build/plugins/indexes/version-records#sboms).

A version that ran in the period and was upgraded or removed since gets what the cluster keeps of it, and an SBOM when the cluster still holds its artifact.

`--out` is a new or empty folder, or a file ending in `.zip`. The tenant's admins may export; every export is in the chain (`evidence.export`). The bundle records what the platform did; whether that meets a law, a contract or a standard is for you and your advisers to judge, and the bundle says so.

### A bundle made without the content permission

Whoever may export evidence and has no content permission, such as the built-in auditor, gets the same files, with what would confirm a guess of what a job read or wrote keyed or left out. A [digest](https://docs.oz0.ai/govern/audit-trail/records#what-a-job-read-or-wrote) in an entry's detail, the call binding of an entry or a gateway record, a gateway record's digests of a model call and an envelope block's digest of its input are HMACs under a key drawn for this one bundle and thrown away, and the salts, which would confirm a guess of a private part, are left out, with the blocks' MACs.

Equal digests stay equal within the bundle, so the two-way check and the origin verdicts are worked out again from the keyed digests and come to the same answers. An exporter the tenant's walls apply to gets such a bundle too, with the private parts the walls leave open.

Some things can no longer be worked out from such a bundle: an entry's actor, target and detail against their hashes, the content hash of an entry with a call binding, a record's digests against its digests hash, the body of a record with a call binding against its content hash, a block's hash, and which entries a child job from before start records listed them inherited, so its verdicts are not re-derived.

`evidence verify` says so, and those rest on the manifest's signature, which it checks. That is why it fails on such a bundle without `--key`, as `audit verify --file` does on a blinded export; `--unpinned` checks it all the same and shows only that it is whole. The bundle's `manifest.json` counts what is keyed, under `keyed`, and the check holds the bundle to those counts.

A gateway record with a call binding is marked `binding_keyed` as well, a redacted one too, and its body has the binding keyed and no `digests_hash`: everything else the body holds is in the line, so the content hash the edge signed would otherwise confirm a guess of the binding, and with it of the step's input.

The record's content hash, its link and the edge's signature of that hash stay as they were and are checked; that the body is the one the edge signed rests on the manifest's signature. A record without a binding keeps its body as the edge wrote it. An entry's ID is the same in every answer, so the IDs of entries from before they were keyed can still have a guess held against them.

### Checking a bundle

`oz0 evidence verify PATH` checks in this order and stops at the first failure:

1. The manifest's signature with the audit key. An unsigned manifest fails.
2. Every file's size and SHA-256, including whether any file is missing or added.
3. The checkpoint's signature, using the same key as the manifest.
4. Every audit-segment entry from the preceding link to the checkpoint's end, for each entry version.
5. Every gateway record and its signature, including whether each redacted record is listed by the erasure that redacted it.
6. The two-way check.
7. The [authority envelopes](https://docs.oz0.ai/concepts/run-receipts/verify#checking-them-offline): each block's hash and link back to block 0; whether each hop only narrows its parent (a widening is a finding); whether each entry's envelope belongs to its job tree (`envelope_mismatch` otherwise); whether each model call, tool call and harness session names a binding of the right form; and the share of audited actions with a complete envelope chain to the person, team or service they ran for.

The verifier opens no connection. The e2e test runs it where only loopback answers. Exit status `0` means all checks passed and found nothing, `1` means a check failed or found a difference, and `2` means the bundle cannot be read.

Pin the audit key with `--key`: the fingerprint `oz0 audit key` prints, given to the auditor some other way than the bundle (a call, a signed letter). With it, a bundle passes only when that key signed it. Without it, `verify` trusts the key the bundle carries and says so: anyone can make a bundle that verifies with its own key.

```text [Output]
ok: tenant acme
  audit key   3f9c… (pinned)
  files       24 checked against the signed manifest
  audit       1840 entries chain correctly (12 redacted, 0 with their private part withheld)
  gateway     611 records chain correctly and their signatures verify (4 redacted)
  two-way     204 jobs checked: 598 model steps by digest, 0 by presence; 611 gateway records, 611 calls the runtimes recorded; 0 finding(s)
  envelopes: 1402 of 1402 audited actions (100.0%) have a complete chain to a subject, 1290 of them to a person or team; 0 widening(s)
  call bindings: 1210 of 1210 model calls, tool calls and harness sessions name a call binding; 0 name none (an operator's call, or a node's runtime from before bindings); the digests are covered by the entries' hashes, not recomputed offline
```

### The gateway's own record

Every model call a node makes goes through the edge's gateway, which holds the provider keys.

The gateway keeps a record of each call in a separate chain per tenant, beside the audit chain. Only edges write it. Each record contains:

- The job, step and attempt the call names, and the node, provider and model.
- The status, including a gateway refusal for a kill switch, budget, quota, data rule or missing key, or a provider that never answered.
- Whether the call streamed and whether the stream ended.
- The SHA-256 and size of the request as forwarded and the response as received. These are computed as the bytes pass, event by event for a stream.
- The tokens and cost.

No prompt or answer: the record holds what metering holds and two digests.

The edge signs each record with a gateway key and waits up to five seconds for it to be written before the response to the node ends; past that the response ends and the record stays queued.

If the database stays away so long that 10,000 records wait, further records are lost rather than holding calls: the edge logs each loss, and the two-way check then reports those calls as `node_only`.

A node-local model's calls never reach the edge and are not in it; the runtime marks them local, and the edge meters each from the node supervisor's report (a `run.model_call` entry with provider `local`). A call the node's supervisor answers itself (no edge reachable, every edge fenced, a request too large) is marked too, and no gateway record is expected of it.

### The two-way check

The gateway's records are one witness of each model call; the job's history is the other: what the runtime says it called, read by the edge from Temporal with the tenant's keys when the job ends (`run.calls`).

The runtime hashes each request it sends the gateway and each response it gets, and its model steps carry the digests.

The check compares the two, job by job and call by call, for the jobs that started and ended in the period (without `--from`, every ended job).

The runtime sends the headers naming a call's job, step and attempt. It also marks recorded calls as local or answered by the node. The check does not take these claims on trust:

- A call under no step, no attempt, a step that makes no model calls or a job that had already ended is a finding.
- A call marked as one the edge did not answer is still compared when the gateway's record shows a complete answer.
- A completed model step needs a gateway record of its attempt, or a local model call the edge metered for it, whatever its calls say.

It reports:

<table>
<thead>
  <tr>
    <th>
      Finding
    </th>
    
    <th>
      What it means
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        gateway_only
      </code>
    </td>
    
    <td>
      The gateway recorded a call the job's history does not account for: an attempt the job never reached, a call its runtime does not list, a job the cluster never heard of (with or without <code>
        --from
      </code>
      
      ), or a job that had ended more than a minute before the call
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        node_only
      </code>
    </td>
    
    <td>
      The history says the runtime made a call the gateway has no record of, or shows a completed model step with no gateway record and no metered local call
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        digest_differs
      </code>
    </td>
    
    <td>
      Both recorded the call, with another request or response
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        unknown_step
      </code>
    </td>
    
    <td>
      A gateway record names a step the job's history does not have
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        wrong_step
      </code>
    </td>
    
    <td>
      A gateway record names a step that makes no model calls, such as a tool call or a hook
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        unattributed
      </code>
    </td>
    
    <td>
      A gateway record of a checked job names no step or no attempt (a harness session's own calls may name no step, inside the session's time), or a node's call names no job
    </td>
  </tr>
</tbody>
</table>

Jobs the period does not hold whole are listed as not checked with the reason, never as findings: they started before the period or before the oldest gateway record retention kept, they ended before the period, their `run.calls` entry is not in the period yet (still running, or not read yet), or their calls came in the last minutes before the export.

A job that runs across the start of a period is not checked by an export with that `--from`, nor by the one before it, which saw it running: export without `--from`, or from before the job started, to check it.

Temporal keeps only a step's last attempt, so calls of earlier, failed attempts are allowed without the runtime listing them.

Some steps are checked by step, not call by call: harness sessions, colleagues' answers, judges, compactions and chat titles, and model steps of runtimes from before this check (a completed one must have a gateway record).

### The audit key and the gateway keys

<table>
<thead>
  <tr>
    <th>
      Key
    </th>
    
    <th>
      Held by
    </th>
    
    <th>
      Signs
    </th>
    
    <th>
      Rotate
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Audit key
    </td>
    
    <td>
      Management, sealed with the master key
    </td>
    
    <td>
      Checkpoints and bundle manifests
    </td>
    
    <td>
      <code>
        oz0 audit key rotate
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Gateway key
    </td>
    
    <td>
      The edges, sealed like their other keys (to the secrets key when the roles run apart)
    </td>
    
    <td>
      Each gateway record
    </td>
    
    <td>
      Every 30 days by itself (the old key is retired then), or <code>
        oz0 audit key rotate --gateway
      </code>
    </td>
  </tr>
</tbody>
</table>

`oz0 audit key` lists both kinds with their fingerprints, retired ones included: what a retired key signed still verifies. Cluster admins rotate; every rotation is audited. Back up the master key: it is what opens the audit key. Edges never read the audit key's sealed half, and management takes only gateway keys from them. A tenant's members see when the newest of each was made in their [encryption status](https://docs.oz0.ai/fleet/temporal-backends/encryption#encryption-status-and-key-rotation), never the keys or their IDs.

### Retention, holds and erasure

The gateway's records follow the audit chain: retention deletes them with the audit entries they sit beside and moves the chain's base, a legal hold keeps them, and erasing a data subject blanks the digests of the subject's calls while the chain and the signatures still verify. The erasure's entry lists the records it blanked (`gateway_redacted_seqs`), and a blanked record no erasure lists fails verification. An export from the chain's base after retention leaves out jobs that started before the oldest record kept.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.