---
title: "Jobs and durability"
description: "Why a job survives crashes, restarts and lost nodes, and what that means for your agents."
canonical_url: "https://docs.oz0.ai/concepts/jobs"
---
# Jobs and durability

> Why a job survives crashes, restarts and lost nodes, and what that means for your agents.

For: cluster operators and plugin authors.

A job is one run of an agent, implemented as a Temporal `AgentWorkflow`. Its job ID is the workflow ID. Temporal holds the job's status, input, answer and complete history; no separate system stores the job.

## What durable means here

- The agent's loop is workflow code. Temporal records every step in the job's history.
- Every model call is an activity. It streams, has a ten-minute timeout and up to five attempts, and heartbeats while it runs. A crashed node loses at most the call in flight.<br />

Temporal is its only retry layer: the provider SDKs do not retry on their own, so an attempt is one call to the provider, and it waits as long as the provider's `retry-after` asks. An error no retry can fix, such as a bad request or a missing key, fails the job at once.<br />

A spent budget waits for a person's decision when a person started the job, and fails at once otherwise, unless the agent's `on_budget` says differently ([budgets that wait](https://docs.oz0.ai/build/parts/contracts-and-limits/budgets#budgets-that-wait)).
- Every tool call is an activity on the home node's queue. Its result is recorded, so it is not called again on replay. Tools can have side effects, so a failed tool call is not retried, unless its function is read-only (MCP's `readOnlyHint`), or idempotent and its server honours the call's idempotency key: those run again, up to three times ([tools](https://docs.oz0.ai/build/parts/tools/call-lifecycle#cancels-time-limits-and-the-idempotency-key)). Otherwise the error goes back to the model, which decides what to do.
- Workflow code reads nothing of the world itself. Network, files, the clock, randomness and the environment are reached only from activities, whose results the history records, so a replay on any node makes the same decisions. The project's lint checks this on every change, and the workflows run in Temporal's workflow sandbox.<br />

A runtime upgrade does not break the jobs already running: every new step is added behind a patch marker, and histories recorded during development, for every kind of job a node runs (agent, flow, tool-call, judge, event, continuation and shadow jobs), are replayed in the project's tests, which fail on a step added without one.<br />

The lint cannot tell a new step from an old one by itself; that half rests on those replay tests, and holds for the steps their histories reach.
- Stopping the client does not stop the job. `oz0 run` waits for the answer, but if you press Ctrl+C the job keeps running. Follow it with `job get`.

If the server restarts, Temporal replays the recorded workflow history. In-flight activities follow their retry or recovery rules. Tool activities run on the job's home node and wait if it is unavailable; recovery of a lost harness session can select a new home.

A lost harness session does not resume its conversation. If the agent is restart-safe, a fresh session starts from job input; otherwise the job waits for a person's `restart` approval. Review may be needed when the lost session could have made effects the platform cannot verify. See [restart waits](https://docs.oz0.ai/agents/approvals/recovery#restart-waits) and [harness restart safety](https://docs.oz0.ai/build/parts/harnesses/recovery#restart-safety).

## IDs and trees

Top-level jobs get an ID such as `job-bb270cf1-b194-4bf1-a0e3-267e061e416a`, or the one you pass with `--id`. The same request under the same ID answers with the job it started instead of starting a new one, which makes retries from apps safe; [any other start under that ID is refused](https://docs.oz0.ai/agents/run-jobs).

Children carry their parent's ID: `<parent>/<agent>-<8 hex>`. That is how `job get` finds a job's tree and how cost is added up per branch.

## Limits and contracts

The platform enforces limits and contracts in workflow code, so a model cannot talk its way around them:

- `max_steps`, `max_tokens` and `timeout` end a job when reached.
- An input that does not match the agent's `input_schema` fails the job before it starts.
- An answer that does not match `output_schema` goes back to the model with the reason, at most twice.

See [Contracts and limits](https://docs.oz0.ai/build/parts/contracts-and-limits).

## History

Every job has a full history in Temporal: each model call with its request and response, each tool call and each child. The edge reads it as the job's steps, with what each one cost: `oz0 job trace`, and the job's page in the console ([Job history](https://docs.oz0.ai/agents/history)). In dev mode the Temporal UI shows the history itself in a browser. History stays as long as the namespace keeps finished workflows; usage rows for cost stay longer.

Payloads larger than 256 KiB, such as a large input or a long conversation that every model call carries, do not go into the history itself: the node stores them on the edge, which keeps them per tenant in the database, and the history holds a reference. So a job can take inputs of many megabytes, up to 32 MiB a payload, where Temporal alone would refuse anything over 2 MB. The edge deletes a payload 90 days after a run last stored it. The Temporal UI shows such a payload as its reference.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.
