Search, compare and fix

For: People investigating jobs and agent changes.

Use console search to find a record or action. Compare jobs to investigate different outcomes, or compare agent versions to identify a changed definition. Findings link to the action that addresses them.

Search everything

Press ⌘K (Ctrl K on Linux and Windows) on any page of the web UI, or click Search at the top of the sidebar. Type a few letters and the box lists what matches, best first, grouped by kind:

Page search terms

Search finds every console page and its tabs using familiar words: budget, egress, allowlist, sso, oidc, mfa, api key, model key, retention, legal hold, works council, four eyes, join token.

A result opens where the thing lives, its tab open, and names the sidebar's category it is in (Govern), with the page for a tab (Govern › Policy).

A page or a category that was renamed is found by its old word too: governance finds Privacy, operate the pages that were under it (Overview, Jobs, Approvals and Queues), quality those that were under it (Findings and Errors), label policies Labels; a category's own word (run, improve) finds every page in it; and a thing that moved to its one home is found by the words of its old place: budget and tenants finds Budgets, teams and quotas Access › Teams and Budgets › Quotas.

With nothing typed, the box lists the pages, in the sidebar's order.

Arrows move, Enter opens, Escape closes. A word matches the start of a name best, then the start of a word in it, then anywhere in it, and last as its letters in order, so bldr finds builder-1. Every word must match.

On PostgreSQL the jobs' words are found through an index, so the search stays quick with millions of jobs: about 3 ms for a word over a million jobs of a tenant, where reading them all took 0.8 s. With the pg_trgm extension (a trusted extension, which the server creates when its database user has the CREATE privilege on the database) a word matches anywhere in a job's text, as everywhere else.

Without it the server falls back to PostgreSQL's own word index, and a job's word must start with the word you type: time finds timeout, out does not; a rare word is as quick, and a word in a tenth of the jobs takes about 1.5 s at a million.

SQLite, in dev mode, reads every job.

The search shows only what your role in the tenant lets you see or do (roles). Each kind appears only to those who may list it on its own page:

• Jobs, as results or as an action's target, require jobs.read. A custom role may leave this out while reading the rest of the tenant.
• A reader finds the pages, tabs and sections a reader has, with no actions.
• An owner finds People and Sign-in methods.
• Only those who read the tenant's secrets find Credentials.
• An operator gets retry and cancel; an admin gets block and new routine.

An action that changes something asks first, in the box: Enter again does it, Escape goes back.

The CLI has the same search:

oz0 search builder
oz0 search "block node-7"
oz0 search timeout --kind job
oz0 search LoopDetected --kind job      # the jobs a loop stopped (on_loop)
oz0 search "possible_loop" --kind job   # the jobs whose trace found one

A job's findings and error are in its words, so the jobs that waited at a loop or at their tree's budget are found as above, and the comparison of two runs of an agent shows which one looped, and what the policy did (loops).

Compare two jobs

oz0 job compare <job-a> <job-b>
oz0 job compare <job-a> <job-b> --content    # also what each was given and answered

               job-2351…                          job-2351…/helper-f248
  agent        lead                               helper
  status       completed                          completed
  version      0.1.0@c16544d3dc31                 0.1.0@c16544d3dc31
  duration     23.1s                              15.3s
  cost         $0.0019                            $0.0005
  ...
Steps:
       A                                        B
       start job start 45ms                     start job start 45ms
    -  model anthropic/claude-sonnet-5-5 110ms
    +                                           tool team.hello 42ms

Two jobs side by side: the plugin and runtime versions they started on, status, duration, cost, model and tool calls, tags and findings, and their steps aligned the way a diff aligns two files. A step is the same in both when its kind and name are (a call to the same model, the same tool function, the same hook, a child job of the same agent); ~ marks one that ended differently (one failed, or needed more attempts), - a step only A has and + one only B has.

Waits for a node are time, not steps, and are left out of the alignment.

In the web UI, a job's page has three ways to compare under More: with the latest other job of its agent, with a job you name, or the versions of its agent. The comparison is the Jobs tab of Improve › Compare, at /compare?a=<job>&b=<job>; each step opens on its job's page. The older address /compare/jobs?a=... opens the same.

Jobs are read from Temporal. A job past the namespace's retention, which Temporal no longer has, is compared from its summary instead: its status, versions, duration, cost, calls and findings, and its steps by kind and name, each kind and name one row with how many there were (x3), how long they took together and how many failed.

Once one side comes from a summary, both sides' steps are compared that way. The comparison says which side came from where (From: its summary); a job with neither a history nor a summary is not found. A job's input and answer go with its history, so with --content a side from a summary has neither.

Their input and answer need the content permission, and you ask for them (--content, or With input and output in the web UI). Each comparison with content is in the audit log as job.compare, once for each job, before anything is shown. Without it you compare their shape.

Compare two versions of an agent

oz0 insights versions triage                         # the newest version that ran, against the one before
oz0 insights versions triage 1.2.0@4f1c09a2b7e3 1.3.0 --diff

For two versions of an agent's or a flow's plugin: the jobs each ran in the last 30 days (--period) with their success rate, failures, p50 and p95 duration and cost per job, marked better or worse against the other; their error groups; their quality gates' evals, case by case (gates); and the plugin's files that changed, with a diff of each text file. Name a version as job summaries do (1.2.0@<commit>), by its version alone, or by the start of its commit.

In the web UI: the Versions tab of Improve › Compare, at /compare?tab=versions&agent=<agent>, or from an agent's deep dive (Versions tab) and a job's page. A file opens read-only at the version a job started on, from a finding's Open the agent's file.

The plugin's files are its source, a private repository's too, so they need the content permission, as what jobs read and wrote does: admins and operators have it, and a tenant's owner can give it to a reader. Without it the comparison shows the numbers, the error groups and the evals, and not the files, and a file does not open.

A comparison reads at most 64 MiB of each version's artifact, unpacked: a larger plugin compares by its numbers alone. Its diffs stop at 512 KiB of text, and at a fixed amount of work: the files after that are listed as left out, without their diff. A server runs at most four comparisons of versions at once; the next waits up to 10 seconds for one to end, and is then refused as busy (try again in a moment).

The server records every version a tenant installs, or tests as a candidate, with its artifact, so versions can be compared after the plugin has moved on. Versions installed before this was recorded are known by their jobs alone: their numbers compare, their files do not. An artifact that no installed version, rollout, gate or retained job uses any more is collected after a week; a version whose artifact was collected compares by its numbers alone, and says so.

The version a job started on

When a job starts, it records the version of its agent's plugin it is routed to (a rollout's candidate on its roll queue, the installed version otherwise) and the runtime version new jobs start on, in its memo. A job that delegates records them for its child jobs. job get and the job's page show them (Plugin version), and its summary, the insights, the error groups and the comparisons use them, so a job summarized after its node was upgraded still counts for the version it ran.

Jobs started before versions were recorded fall back to the version their node ran when the job was first summarized.

Jobs that replay recorded answers

A job that replays recorded model answers (the studio's test bench) or recorded flow steps (a flow's evals) is not a real outcome. Its summary says so, and the KPIs, the deep dives, the error groups and the comparison of versions leave it out; the search, the Findings page and the job's page label it replay.

Findings that act

A finding on a job's Steps tab, on the Findings page (in a job's dialog), or in a comparison offers its fix, where there is one:

waiting_for_person is new: an approval or a question that has waited 5 minutes for a person.

Run it again starts the job once more with the same agent, input, tags and budget, under a new ID, through the same checks as run (kill switches, quotas, the input's contract), and is in the audit log as job.retry. Operators and above may; the toast links the comparison of the two.

A routine's fire runs again from its routine, a child job with the job that delegated to it, and a job of a data subject with its subject. A conversation's turn runs again as the conversation's next turn, with the same message and earlier turns (also when it is about a data subject), once it is the latest.

A webhook's job runs again when its sender delivers again: it ran as work from outside, on sandbox tier 1, and a run from here would be work from inside. An input too large for Temporal's history is read back from where the server keeps it; when that is gone, the job does not run again with less.

The new job holds what the old one held: it starts with the old job's compartments and level, so the walls that close the old job to someone close the new one, and a person the walls close the old job to may not run it again.

oz0 job retry <job-id>

The buttons follow your role: running a job again and cancelling it need an operator; the others are for everyone who sees the job.