---
title: "Job history"
description: "Every step of every job, as a trace you can read from the CLI and the web UI, with findings about what looks wrong."
canonical_url: "https://docs.oz0.ai/agents/history"
---
# Job history

> Every step of every job, as a trace you can read from the CLI and the web UI, with findings about what looks wrong.

For: people running and reviewing agents.

Inspect a job's steps to see its model calls, tool arguments and results, hooks, approvals, children and failures. Temporal records that complete history; the edge reads it and presents the job's steps.

## A job's steps

```bash [Terminal]
oz0 job trace <job-id>
```

```text [Output]
Job:     job-f60d03cb-c410-4352-a3b4-8ee042b805aa (lead), completed after 1.5s, $0.0024
Trace:   2e156de40610647ba330ff84c01601f4 in your tracing backend

AT      TOOK   KIND   STEP                                     NODE               TOKENS  COST     STATUS  ID
+197ms  33ms   start  job start                                node-d5fba02a1365  -       -        ok      activity-1
+324ms  105ms  model  anthropic/claude-sonnet-5-5              node-d5fba02a1365  196/20  $0.0006  ok      activity-2
+555ms  652ms  child  helper (job …/helper-e2c82af1)           node-8eda3997fd54  307/44  $0.0010  ok      child-23
+1.3s   101ms  model  anthropic/claude-sonnet-5-5              node-d5fba02a1365  269/25  $0.0008  ok      activity-3
```

Each row is one step, in the order they began:

<table>
<thead>
  <tr>
    <th>
      Kind
    </th>
    
    <th>
      What it is
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        queue
      </code>
    </td>
    
    <td>
      The job waited at least a second for a node to take it, or to take its next step
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        start
      </code>
    </td>
    
    <td>
      The job's start on its home node, which reads the hooks and approvals that apply to it
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        model
      </code>
    </td>
    
    <td>
      One model request, with the tokens and the cost of all its attempts, as the gateway metered them
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        tool
      </code>
    </td>
    
    <td>
      A tool call, on the node it ran on, with what its plugin <a href="https://docs.oz0.ai/build/parts/tools/model-calls#report-what-your-own-services-cost">
        reported
      </a>
      
       for the services it called. A harness session's tool calls are listed inside the session
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        hook
      </code>
    </td>
    
    <td>
      A <a href="https://docs.oz0.ai/build/parts/hooks">
        hook
      </a>
      
       and what it decided: <code>
        allow
      </code>
      
      , <code>
        deny
      </code>
      
      , <code>
        modify
      </code>
      
       or <code>
        escalate
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        approval
      </code>
    </td>
    
    <td>
      A wait for a person, who decided and how, or <code>
        timeout
      </code>
      
       when nobody did
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        child
      </code>
    </td>
    
    <td>
      A child job, with what its whole branch cost. <code>
        job trace
      </code>
      
       on its ID shows its own steps
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        harness
      </code>
    </td>
    
    <td>
      A <a href="https://docs.oz0.ai/build/parts/harnesses">
        harness
      </a>
      
       session, with every model call it made
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        flow
      </code>
    </td>
    
    <td>
      A <a href="https://docs.oz0.ai/build/parts/flows/flow-tools#flows-as-tools">
        flow tool's
      </a>
      
       call: its flow ran as a child job, and its steps are listed under it
    </td>
  </tr>
</tbody>
</table>

The time a step waited for a node is the gap between when it was asked for and when it started. A model call that was tried again shows its last attempt and why the one before failed. While a job runs, `job trace` shows what runs now, and how long it has run.

`job trace --json` prints the same for scripts.

### What looks wrong

Below the steps come findings: rules over the trace that point at what is worth a look.

<table>
<thead>
  <tr>
    <th>
      Finding
    </th>
    
    <th>
      When
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        idle
      </code>
    </td>
    
    <td>
      Nothing ran for ten minutes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        waiting_for_node
      </code>
    </td>
    
    <td>
      The job or a step waited more than a minute for a node
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        waiting_for_person
      </code>
    </td>
    
    <td>
      An approval or a question has waited five minutes for a person
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        repeated_tool
      </code>
    </td>
    
    <td>
      One tool was called ten times or more
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        possible_loop
      </code>
    </td>
    
    <td>
      One tool was called three times with the same input (<code>
        loops.repeats
      </code>
      
      ); it says what the agent's <a href="https://docs.oz0.ai/build/parts/contracts-and-limits/limits#loops">
        <code>
          on_loop
        </code>
      </a>
      
       did: the job waits for a person, who continued or stopped it, the policy stopped it, or it only reported
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        repeated_error
      </code>
    </td>
    
    <td>
      Three steps or more failed the same way
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        retries
      </code>
    </td>
    
    <td>
      Three steps or more needed another attempt, or one step needed three
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        budget
      </code>
    </td>
    
    <td>
      The job has spent 80% of its own <a href="https://docs.oz0.ai/build/parts/contracts-and-limits/budgets#budgets">
        budget
      </a>
      
       (<code>
        limits.max_cost_usd
      </code>
      
      , or <code>
        --budget
      </code>
      
      ; the share is <code>
        spend.warn_share
      </code>
      
      ); once it is spent, its next model call fails the job with <code>
        BudgetExceeded
      </code>
      
      , or makes it wait for a person when its agent says <a href="https://docs.oz0.ai/build/parts/contracts-and-limits/budgets#budgets-that-wait">
        <code>
          on_budget: wait
        </code>
      </a>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        workflow_failures
      </code>
    </td>
    
    <td>
      The job's workflow code failed on a node, for example after a change that broke a running job
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        long_run
      </code>
    </td>
    
    <td>
      The job has run for more than half an hour
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        slow_step
      </code>
    </td>
    
    <td>
      A model call, tool call or hook took more than five minutes
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        model_fallback
      </code>
    </td>
    
    <td>
      A model was replaced by another: a fallback answered, or an alias's resolution left models out (<a href="https://docs.oz0.ai/spend/models-and-keys/aliases#no-model-is-replaced-silently">
        no model is replaced silently
      </a>
      
      )
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        model_override
      </code>
    </td>
    
    <td>
      The tenant's <a href="https://docs.oz0.ai/spend/models-and-keys/mapping#map-a-plugins-models">
        mapping rule
      </a>
      
       chose the agent's model when the job started; it names the rule and the model (info)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        effort_adjusted
      </code>
    </td>
    
    <td>
      The <a href="https://docs.oz0.ai/spend/models-and-keys/effort-and-cascades#reasoning-effort">
        reasoning effort
      </a>
      
       asked for was changed when the job started: lowered to the tenant's ceiling, rounded to a level the model takes, or not sent to a model that takes none (info)
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        judge_same_model
      </code>
    </td>
    
    <td>
      A judge ran, or was about to run, on the model of the work it judges: an outcome check's judge or reviewer, or a best-of's judge, which stops the job before it is asked (<a href="https://docs.oz0.ai/improve/outcome-checks/judges#a-judge-on-another-model-than-the-agent">
        a judge on another model
      </a>
      
      )
    </td>
  </tr>
</tbody>
</table>

Findings themselves only report. What acts is the agent's policy, enforced in the runtime and at the gateway: a loop waits for a person or stops the job (`on_loop`), useless results leave the agent only its answer, and a spent budget waits for a person (`on_budget`); the findings say what happened. Where there is a fix, such as running the job again or opening what it waits for, the finding offers it to a person ([findings that act](https://docs.oz0.ai/improve/search-and-compare#findings-that-act)).

### Across jobs

The edge also keeps a summary of every job in the management database: its agent, status, times, steps, calls, tokens, its own cost, its error and its findings. It writes the summary when the job ends, and every 30 seconds while it runs, so a looping job shows up before it is done. Summaries stay after Temporal has forgotten the job.

```bash [Terminal]
oz0 job findings                       # jobs with a warning, the last day
oz0 job findings --code possible_loop --since 168h
oz0 job findings --code all --agent lead --json
```

The console's **Findings** page, under *Improve*, shows the same with dropdowns for the finding (any warning when none is chosen) and the agent, and the [period](https://docs.oz0.ai/getting-started/web-ui/navigation#periods-filters-and-pages) (the last 24 hours unless you pick another), a page at a time. Open a job there to see all its findings and go to its steps.

### What a step read and wrote

A trace shows the shape of a job: names, times, tokens and cost. What a step read and wrote can be your customers' data, so it is shown one step at a time, to those allowed to see it:

```bash [Terminal]
oz0 job step <job-id> activity-2
```

That prints the step's input and output: a model call's messages and its response, a tool call's arguments and result, what a person was asked and what they answered. Payloads kept in [the edge's storage](https://docs.oz0.ai/concepts/jobs#history) are read back for it. It needs the permission to see the tenant's content, which operators, admins and owners have and readers do not, unless an owner says otherwise ([roles](https://docs.oz0.ai/getting-started/web-ui/accounts#accounts-and-roles)). Each look is recorded in the audit log, as `job.step.read`:

```bash [Terminal]
oz0 audit --action job.step.read
```

Without that permission, the answer, the live stream's text, tool arguments and results, and approvals' details are left out everywhere: in the console, `job get`, `job watch` and the runtime API.

The trace still tells steps that got the same input apart from the others, by a digest made for each answer alone, which nothing outside that answer compares with: no guess of a step's input can be checked against it.

A comparison of two jobs without content (`job compare`, `CompareJobs`) keys both jobs' steps with one such digest, so the same input still lines up across the two. An offloaded result's digest is left out, and a read of that result is named by a digest made the same way.

The same holds for a job walled off from you by a label policy, and for a turn of someone else's private chat.

### A flow tool's steps

An agent that calls a [flow tool](https://docs.oz0.ai/build/parts/flows/flow-tools#flows-as-tools) decided only to call it; its flow's steps ran in a child job. The agent's trace shows the call as one `flow` step and lists the flow's steps under it, each as it is in the flow's own job, with what it cost and who approved it:

```text [Output]
AT      TOOK   KIND      STEP                                    NODE               TOKENS  COST  STATUS  ID
+1.2s   4.1s   flow      greeter.hello (job …/hello-3f2a9c1e)    node-d5fba02a1365  -       -     ok      child-17
+1.3s   20ms   start       └ job start                           node-d5fba02a1365  -       -     ok      activity-1
+1.4s   35ms   tool        └ team.hello                          node-d5fba02a1365  -       -     ok      activity-2
+1.5s   3.6s   approval    └ team.note: approved by user:kim     -                  -       -     ok      approval-0d4e
+5.1s   30ms   tool        └ team.note                           node-d5fba02a1365  -       -     ok      activity-4
```

A nested step's ID is its ID in the flow's job, which `job step` reads from there: `oz0 job step <flow job> activity-4`. The console's job page shows the flow tool as one row that opens to its steps. A trace nests the steps of at most 20 flow tool calls; `job trace` on the flow's job shows any of them on its own. The audit log has the call as `run.flow_tool` on the agent's job and each step as its own entry on the flow's job, with `flow_tool` and `parent_job` in its detail.

### In your tracing backend

When the edge [exports traces](https://docs.oz0.ai/fleet/observability#traces), `job trace` names the job's OpenTelemetry trace. The deep dive and the trace show the same job: the deep dive knows the waits, the approvals and the cost, and your tracing backend keeps it as long as you keep traces.

## In the console

A job's page in the [console](https://docs.oz0.ai/getting-started/web-ui) shows the same steps as a waterfall, with the findings above it. Each bar starts when its step was asked for: the dashed part is time spent waiting for a node, and the solid part the step itself.

Click a step and its details open under its own row, as an event's do in Temporal's history, so the first step of a long trace is read where it is: when it began, how long it waited and ran, its node, attempts, tokens and cost, and what it decided. Admins can open its input and output there, which the audit log records too.

The page also shows the answer or the error, the live stream while the job runs, the tree of child jobs with the cost of each branch, and a link to the job's history in the Temporal UI.

## Other ways in

```bash [Terminal]
oz0 job list
oz0 job watch <job-id>        # follow a running job live
oz0 job get <job-id>          # status, answer or error, children, cost per branch
oz0 job get <job-id> --json   # everything, for scripts
```

### In the Temporal UI

For embedded or external self-hosted Temporal, the server's management role serves Temporal's console at `https://<server>:8443/temporal/`, behind the [console](https://docs.oz0.ai/getting-started/web-ui)'s sign-in and for admins only, since it can change any job. A job's page in the console links straight to its history there. Temporal Cloud uses its own UI instead.

In dev mode Temporal's console also runs on port 8233 without a sign-in, for the local machine (`oz0 dev` prints its address).

Open the tenant's namespace, `default` unless you created others, and find the job by its ID. Each activity's summary says what it is: `tool acme_github.merge`, `hook pre_tool_call acme_policy.check`, `job start`, `harness claude-agent-sdk`, and `request model: …` for model calls. Child workflows are delegation.

## How long history stays

History stays as long as the namespace keeps finished workflows, and so do traces and steps. `job get` can show a job's answer only while Temporal still has it. Usage rows for cost and job summaries stay in the management database longer.

<span id="in-the-web-ui">

in the web ui

</span>

: [Read the section](https://docs.oz0.ai/agents/history#in-the-console).


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.