Errors and insights
For: People diagnosing recurring failures.
Start with grouped errors to find recurring failures, then open the affected agent, tool or node. Use the job's trace for the content behind an individual failure. Queues explains jobs that are waiting.
The edge writes job summaries when a job ends and every 30 seconds while it runs. Management can read those summaries independently, after Temporal has removed the job.
Summaries contain failure kinds, steps and attempt counts. They exclude job content because they outlive the job and are read without its labels. Prompts, answers and error text remain on the job and its trace, subject to content permission.
Errors, grouped
oz0 job errors # the last 7 days
oz0 job errors --tool acme_github --since 24h
oz0 job search group:4f1c09a2b7e3 # every job with one of them
Failures of jobs, and of their steps (a tool call, a model call, a hook), are grouped by a fingerprint of the message with what varies taken out: IDs, numbers, quoted values, URLs and addresses. timeout after 30s calling https://api.example.com/x?id=3 and timeout after 31s calling https://api.example.com/y are one group. The words only decide the group: it shows the failure's kind and step, not the words, and its fingerprint is keyed per tenant, so that it confirms no guess of them.
Open one of its jobs to read what failed. A tool's or a hook's errors are grouped per function; a job's errors and a model's are grouped across agents and models, so one outage is one row. Each group says how often, in how many jobs, when it was first and last seen, and which agents, nodes and versions it hit: for a tool's or a hook's errors the version of the tool's plugin, for the others the agent's.
A step that failed and then worked on a later attempt counts too.
The console's Errors page lists groups for the period you pick, or the last 7 days by default. It shows the groups with the most jobs first, 25, 50 or 100 a page, with dropdowns for the agent, tool and node.
Counts are fixed at the moment the first page was read, so later pages neither repeat nor pass over a group. The page is read again every 15 seconds: page 1 gets new counts, while later pages keep the first page's snapshot. Go back to page 1 for current counts.
Every job with it, in a group's dialog, searches the Jobs page for the group over the same period.
KPIs and deep dives
The Overview shows the tenant's KPIs over the period you pick (the last day, week or 30 days, this month, or a range of days), each against the period before: jobs, success rate (completed out of those that ended; a canceled job counts neither way), p50 and p95 duration of finished jobs, cost (every metered call, child jobs included), and approvals waiting now (as the Approvals page lists them).
Below them, Trends draws jobs, duration and cost over the period, each chart with a Table button for its numbers; then the period's jobs, filtered by status and agent.
oz0 insights overview --period 168h
oz0 insights agent acme-triage
oz0 insights tool acme_github # every function; or acme_github.search
oz0 insights node node-7
A deep dive shows one agent, tool or node over a period: its runs (jobs for an agent, calls for a tool, steps for a node), failure rate, p50 and p95 latency and cost per day or hour, against the period before; by version (the agent's or tool's plugin version that ran, or the runtime of the jobs a node ran); what it used or ran (an agent's tools, models and nodes; a tool's functions, agents and nodes; a node's agents and tools); its most common errors; and its latest jobs.
In the web UI, open one from an agent's or a node's dialog, or a tool's name: /agents/<name>, /tools/<id>, /nodes/<id>. Its tabs are Trends, Jobs, Errors, Versions and Uses (Callers for a tool, Workload for a node); an agent's Scorecard, Evals and Permissions are under More.
The KPIs, deep dives and insights are per tenant, agent, tool, node and version, never per person: none of them is a figure about people, so works-council mode leaves them as they are, except the scorecard's figures made of people's decisions (below).
An agent's scorecard
An agent's deep dive has a Scorecard tab, per version and over the period's days or hours, from the same jobs (replays and evals' own jobs left out):
| Figure | What it is |
|---|---|
| Billed cost per successful run | The headline: everything the agent's runs were metered for at the providers' prices, failed runs included, over the runs that succeeded. Fewer tokens is not lower cost: caching moves most of the bill |
| Runs, successes, success rate | Jobs that ended; a canceled job counts neither way |
| Tokens by class | Uncached input, cache reads, cache writes and output, apart, from the usage rows of the jobs and the jobs below them |
| People asked | Approvals and questions per run, and the share of runs a person was asked something in |
| Denied, edited, waited | Of the approvals people decided: the share denied, the share approved with edits, and how long decisions waited (p50, p95). A timeout or a deadline is not a person's decision |
Under the table, Oversight shows the agent's agreement per action class: how often people approved its proposed calls of each class as proposed, over the last 90 days or since its latest drift event, with whether the promotion threshold (95% over 50 decisions) holds. It is data only, through the same rule: a class decided by fewer people than the threshold is hidden. The inventory's deployment profile shows the same with the agent's test approvals caught.
The four figures about people go together, never the intervention rate alone: a rising number of approvals with a falling share denied is the warning sign, and an intervention rate on its own would reward approving everything. The scorecard is per agent and version, never per person or per approver.
While works-council mode is on and fewer people than its threshold decided the agent's approvals in a period, a bucket or a version, all four are hidden there, with a note: the denial rate, the edit rate, the waits and the edits count, and with them the people asked per run and the approvals and questions it is counted from.
oz0 insights agent support --period 720h # the deep dive, with the scorecard and the eval runs
oz0 insights agent support --period 720h --export csv --out support.csv
oz0 insights agent support --export json
The export has a row for the period, one per version and one per day (or hour, for a day), with every figure above; the figures about people are empty (CSV) or left out (JSON) where works-council mode hides them. A scorecard reads at most 50,000 jobs, 200,000 usage rows and 200,000 approval entries of the agent's; past that, it says the figures are short (truncated) rather than showing them as whole.
Export CSV on the Scorecard tab downloads the same. The Evals tab lists the agent's eval runs and its open regression and model-change findings.
The scorecard also shows the agent's open privilege from the tenant's newest privilege replay that scored it under the policy in force (one scored under a proposed change is not shown), to those who may read governance: the share, weighted by action class, of the functions it could call and did not, with the replay it comes from and how many tighter rules that replay suggested. It is absent until a privilege replay has run.
The version is the one each job started on; two versions compare side by side with insights versions. A tool's and a node's latency come from a histogram of each job's step durations, so their percentiles are estimates within a bucket (10 ms up to 15 minutes); an agent's come from its jobs' exact durations.
Search and tags
oz0 run triage "..." --tag incident-42 --tag nightly
oz0 job tag <job-id> +reviewed -nightly
oz0 job search "agent:triage status:failed timeout"
oz0 job search tag:incident-42 --since 720h
A search takes words, which must each be in a job's ID, agent, error or tags, and filters: agent:, status:, tag:, node:, version:, group: (an error group), finding:, and is:top or is:child. A summary keeps a failure's kind and a fixed sentence, not its words. On a summary an older edge wrote with the words in it, a reader without the content permission, or one a wall applies to, matches the ID, agent and tags only, so a search answers nothing about what a failure said.
The Jobs page searches the same way when you type two words or more, or a filter, in its search box, over the summaries of jobs that started in its period; one word without a colon finds the jobs whose ID holds it among Temporal's jobs of the period.
Tags are lowercase letters, digits and . _ : / -, at most 64 characters and 20 per job. Operators and above set them when a job starts or later, also on a job that ended; each change is in the audit log as job.tags. The job list, the job's page and search show them.
If the tags of a new job cannot be written, the job still starts, and the edge logs a warning: add them with job tag. A job ID you choose yourself (run --id) may be started again once its job has ended; that new run starts with only the tags you give it, not the old run's.
Starting it again while it runs is a retry: you get the running job, and the tags you give are added to its own.
Export a job
oz0 job export <job-id> -o job.json # its shape
oz0 job export <job-id> --content -o job.json # with what its steps read and wrote
An export is one JSON document ("format": "oz0.job-export/v1") with the job, its steps, findings and tags: names, times, nodes, tokens and cost. With --content it also has each step's input and output and the job's answer, which needs the content permission and is in the audit log as job.export. An export holds at most 16 MiB of step content; the steps past that are listed in steps_omitted, and the job's page shows each of them on its own.
Every secret of the tenant of 6 characters or more is replaced by [redacted:<name>] wherever it appears, in either kind of export: as it is, inside text that holds JSON (a tool's result written as JSON text, escaped once more for each level, whichever encoder wrote it), quoted in an error's cause, escaped for a URL, in hex, and in base64, also inside longer encoded text such as HTTP Basic credentials, where it follows a user name.
The export's values are redacted before they are written, and the document is searched again after. A secret split across two values is not found. Without the content permission an export's steps carry no digest that a guess of their input could be checked against, as the trace shows them.
Shorter secret values are left as they are, since replacing them would also replace ordinary words: keep tokens and PINs that must not leave in an export at 6 characters or more. The export says how many such short secrets the tenant has (short_secrets in the API; a note from job export, a warning in the web UI).
The job's page in the web UI has the same under Export.
Search, compare and fix
The ⌘K search across agents, tools, nodes, jobs, pages and actions; two jobs or two versions of an agent side by side; the version a job started on; and findings that offer their fix.
Measured savings
Run an agent and one change to it in pairs on the same cases, side-effect free, and see whether the change costs less per successful run at an equal pass rate.