Concepts

Jobs and durability

Why a job survives crashes, restarts and lost nodes, and what that means for your agents.

For: cluster operators and plugin authors.

A job is one run of an agent, implemented as a Temporal AgentWorkflow. Its job ID is the workflow ID. Temporal holds the job's status, input, answer and complete history; no separate system stores the job.

What durable means here

  • The agent's loop is workflow code. Temporal records every step in the job's history.
  • Every model call is an activity. It streams, has a ten-minute timeout and up to five attempts, and heartbeats while it runs. A crashed node loses at most the call in flight.
    Temporal is its only retry layer: the provider SDKs do not retry on their own, so an attempt is one call to the provider, and it waits as long as the provider's retry-after asks. An error no retry can fix, such as a bad request or a missing key, fails the job at once.
    A spent budget waits for a person's decision when a person started the job, and fails at once otherwise, unless the agent's on_budget says differently (budgets that wait).
  • Every tool call is an activity on the home node's queue. Its result is recorded, so it is not called again on replay. Tools can have side effects, so a failed tool call is not retried, unless its function is read-only (MCP's readOnlyHint), or idempotent and its server honours the call's idempotency key: those run again, up to three times (tools). Otherwise the error goes back to the model, which decides what to do.
  • Workflow code reads nothing of the world itself. Network, files, the clock, randomness and the environment are reached only from activities, whose results the history records, so a replay on any node makes the same decisions. The project's lint checks this on every change, and the workflows run in Temporal's workflow sandbox.
    A runtime upgrade does not break the jobs already running: every new step is added behind a patch marker, and histories recorded during development, for every kind of job a node runs (agent, flow, tool-call, judge, event, continuation and shadow jobs), are replayed in the project's tests, which fail on a step added without one.
    The lint cannot tell a new step from an old one by itself; that half rests on those replay tests, and holds for the steps their histories reach.
  • Stopping the client does not stop the job. oz0 run waits for the answer, but if you press Ctrl+C the job keeps running. Follow it with job get.

If the server restarts, Temporal replays the recorded workflow history. In-flight activities follow their retry or recovery rules. Tool activities run on the job's home node and wait if it is unavailable; recovery of a lost harness session can select a new home.

A lost harness session does not resume its conversation. If the agent is restart-safe, a fresh session starts from job input; otherwise the job waits for a person's restart approval. Review may be needed when the lost session could have made effects the platform cannot verify. See restart waits and harness restart safety.

IDs and trees

Top-level jobs get an ID such as job-bb270cf1-b194-4bf1-a0e3-267e061e416a, or the one you pass with --id. The same request under the same ID answers with the job it started instead of starting a new one, which makes retries from apps safe; any other start under that ID is refused.

Children carry their parent's ID: <parent>/<agent>-<8 hex>. That is how job get finds a job's tree and how cost is added up per branch.

Limits and contracts

The platform enforces limits and contracts in workflow code, so a model cannot talk its way around them:

  • max_steps, max_tokens and timeout end a job when reached.
  • An input that does not match the agent's input_schema fails the job before it starts.
  • An answer that does not match output_schema goes back to the model with the reason, at most twice.

See Contracts and limits.

History

Every job has a full history in Temporal: each model call with its request and response, each tool call and each child. The edge reads it as the job's steps, with what each one cost: oz0 job trace, and the job's page in the console (Job history). In dev mode the Temporal UI shows the history itself in a browser. History stays as long as the namespace keeps finished workflows; usage rows for cost stay longer.

Payloads larger than 256 KiB, such as a large input or a long conversation that every model call carries, do not go into the history itself: the node stores them on the edge, which keeps them per tenant in the database, and the history holds a reference. So a job can take inputs of many megabytes, up to 32 MiB a payload, where Temporal alone would refuse anything over 2 MB. The edge deletes a payload 90 days after a run last stored it. The Temporal UI shows such a payload as its reference.

Copyright © 2026