--- title: "Notifications" description: "Tell people and systems when an approval waits, escalates or expires, by email, a signed webhook or a chat channel." canonical_url: "https://docs.oz0.ai/admin/notifications" --- # Notifications > Tell people and systems when an approval waits, escalates or expires, by email, a signed webhook or a chat channel. For: cluster and tenant administrators. Create rules for events that people or systems must act on. Configure the destination and inspect its delivery results. An [approval](https://docs.oz0.ai/agents/approvals) that nobody sees waits until its deadline denies it. Notification rules tell the right people when something waits for them: by email through your own mail relay, by a signed webhook to your own systems, or in a chat channel. ## Rules A tenant's admins keep its rules. A rule names events, may narrow them to agents, tools and risk classes, and sends them to targets: ```bash [Terminal] oz0 notify rule put payments --event approval.waiting --event approval.escalated --tool 'bank.*' \ --email-team payments --email-role admin oz0 notify rule put desk --event approval.waiting --risk destructive \ --webhook https://desk.acme.example/oz0 --secret DESK_HOOK_KEY --decide-key 3f9a1c2b7d4e oz0 notify rule list oz0 notify rule test payments # one test notification to each target oz0 notify rule delete nr-0881cb5609f6 ```
Event When
approval.waiting an approval starts waiting
approval.escalated the escalation ladder takes a step
approval.expired the deadline denied it
question.waiting an agent asks a person
budget_request.waiting a budget request waits
job.failed a job failed
budget.share spending reached one of the tenant's spend alerts of a budget (80% and 100% by default; the 100% alert says whether jobs wait and who decides)
eval.regression an eval routine confirmed a regression (facts only: the eval, the cases that flipped, the finding)
model.changed a model canary saw the model behind a route change
tool_rule.expiring a rule of the tool policy ends within a week (once per rule and end)
secret.expiring a secret's expiry is days away: 14, 3 and 1 by default (once per secret, expiry and day); it also goes by email to the tenant's admins and whoever set the secret, and names the secret, its expiry and the days left, never a value or the setter
incident.deadline a deadline of an incident is near, or passed with its stage not marked (once per reminder point); the owner and the tenant's admins get it by email whatever the rules say
plugin.vulnerable an installed plugin version locks a package with a known vulnerability new to it (once per version and set of advisories; also to the owner persons of its agents, flows and tools and the tenant's admins); --tool filters on the plugin's name
label_source.unlabelled a tool that declares a label source returned results without a level, so the tenant's default level applied ( Taint and labels ): once per function and day, also to the tool's owner person and the tenant's admins; counts per function, never per person
quality.drift a drift finding names its cause for an agent: the model behind a name, the judge on its anchors, the agent's consistency, or the plugin's version (facts only: the cause and the finding; once per finding and evidence)
model.retiring a model an agent pins retires soon : when the notice starts (30 days by default), at seven days and one day left, and on the date (facts: the provider, the model, the date, its successor, the days left)
spend.anomaly a day's spend was far above its usual (the spend-anomaly finding , once per day): the day, its spend and usual, and what rose most by agent, model and plugin; never a team or a person
plugin.custody a refresh of an index finds that an installed plugin's listing moved or that its newest version is in new hands (once per plugin and listing state; also by email to the tenant's admins): the plugin, its listing and where it moved or the new custody epoch; --tool filters on the plugin's name
approval.pattern most decisions in a queue came within seconds of the card being shown (the approval pattern finding , once per finding): the queue (the tenant's or a team's of five people or more), the share and the number of decisions; never a person
approval.flood one job tree or one agent asked a flood of approvals (the flood finding , once per finding): how many, in what time, and the agent or job tree
decision.contested a person contests the decision record of an agent's job (once per contest); it also goes by email to the agent's owner person, and names the agent, the job and the contest, never the person or what they dispute
plugin.retired a plugin was retired (once per retirement): its name and the reason; also by email to the owner persons of its agents, flows and tools, else their teams' admins, else the tenant's admins; --tool filters on the plugin's name
dependency.retired a plugin that others depend on was retired (once per retirement): which plugin and what depends on it; also by email to the owner persons and team admins of the dependent plugins, flows and routines, else the tenant's admins
team.orphaned a team that owns routines or inventory items has no person as a member, or no person as its admin (once per orphaning): the team, why, since when and what it owns; also by email to the tenant's admins. See teams that nobody answers for
connection.reconnect a connection's provider refused to renew its token (a revoked, expired or spent refresh token), so its tools' calls get none until it is connected again (once per refusal): the connection, its plugin and the provider's reason, never a token; also by email to the tenant's admins; --tool filters on the plugin's name
`--tool` takes `` or `.`, with `*` for anything; `--risk` takes `read_only`, `destructive`, `open_world` or `undeclared`. The ladder's own steps reach the people the step names (the people who may decide, the agent's owner, the tenant's admins) by email, whatever the rules say, once the operator has configured email; rules add more targets and the other events. In the console, Settings, Notifications lists the tenant's rules with their latest deliveries. Add notification rule opens the form in a dialog; cancelling clears the draft, and a rejected save keeps it for correction. Each person sets the address their own notifications go to on the same tab. `oz0 notify rule list` shows each rule's latest deliveries with their state, attempts and last error. Rules reach only the tenant's current members: someone who left the tenant, or whose account is disabled, gets nothing, even when a team or an agent's owner still names them. When the agent's owner has left, the ladder's owner step goes to the owner team's admins, then its members, then the tenant's admins, so it never reaches nobody. Operators, admins and owners of the tenant may list its rules; only its admins and owners put, delete and test them. Operators see a webhook's scheme and host only, since its URL may carry a token; admins see the whole URL they edit. Every change is in the audit trail (`notify.rule.put`, `notify.rule.delete`, `notify.rule.test`). ## Email Email goes through an SMTP relay the operator configures for the cluster, and is off until then: ```bash [Terminal] orchestrator-zero server start --smtp-addr smtp.acme.example:587 --smtp-from oz0@acme.example \ --smtp-user oz0 --smtp-password-file /etc/oz0/smtp-password --smtp-tls starttls ``` Mail always goes over TLS: STARTTLS (port 587, the default) or TLS from the start (`--smtp-tls tls`, port 465), and the relay's certificate is verified (`--smtp-ca-file` for a relay with a private CA). The password comes from a file, never a flag. The edge sends the mail; nodes never do. The same relay also carries the mail agents send when a tenant installs [ai.oz0/email](https://docs.oz0.ai/build/plugins/verified/email): the edge sends it too, only to the tenant's own mail domains unless the tenant allows others, with the AI label, and records each message. It goes from `--smtp-from` unless the operator lists the tenant's sender's domain for that tenant with `--smtp-sender-domains =` (repeatable); otherwise the tenant's sender is its `Reply-To`. A notification's mail is not changed by it. A person gets mail at the address their identity provider gives when they sign in, or the one they set themselves (Settings, Notifications, or `oz0 notify address ana@acme.example`; `-` removes it). A password account has none until its person sets one. Rules name people by role, team or account name, never by address. A notification carries the event's facts: the tenant, the agent and its version, the tool, the job's link, the deadline and the ladder's step. It never carries a link or a code that decides anything: the person opens OZ and decides there. It never carries the call's arguments unless the tenant's admins turn that on: ```bash [Terminal] oz0 tenant notify-content acme arguments --reason "the payments desk decides from email" oz0 tenant notify-content acme facts ``` Even then, an email carries the arguments only to a person who could read them in OZ: one who holds the content permission in the tenant, for a job that is not a turn of someone else's private [chat](https://docs.oz0.ai/agents/conversations). Everyone else gets the facts. This is checked when the mail is sent, so a person who lost the permission before a retry gets the facts too. A webhook carries the arguments when the tenant's setting says so, since its admins chose the endpoint. Chat channel targets carry facts. Arguments can hold personal data, which then leaves OZ to the mail relay and to every webhook endpoint. Whether a tenant may send them there is a legal question for your organisation; OZ keeps facts only unless an admin decides otherwise, records who did and why (`tenant.notify_content`), and treats it as a setting that loosens oversight. Text an agent wrote carries the tenant's [AI label](https://docs.oz0.ai/govern/ai-labels). ## Webhooks A webhook target gets a JSON `POST` for each event: ```json { "type": "approval.escalated", "tenant": "acme", "agent": "payer", "tool": "bank.pay", "job": "job-1e71252a-...", "job_url": "https://oz.acme.example/t/acme/jobs/job-1e71252a-...", "approval": "a-5ccd30628bdc", "step": 2, "steps": 3, "to": "owner", "level": "L2", "requested_at": "2026-10-08T21:01:29Z", "deadline": "2026-10-09T21:01:29Z", "at": "2026-10-09T09:01:29Z", "decide": "decide in OZ (oz0 approval approve|deny), or with an API key through the runtime API's DecideApproval; poll ListApprovals for the outcome" } ``` Its headers:
Header What
X-Oz0-Signature sha256= and the hex HMAC-SHA256 of the timestamp, a dot and the body, with the target's secret
X-Oz0-Timestamp seconds since 1970, the time the delivery was signed
X-Oz0-Event the event, such as approval.waiting
Idempotency-Key the same on every attempt of one delivery: act on it once
The secret is a tenant secret, sealed for the edges, set as a signing key bound to the webhook's host (`oz0 secret set DESK_HOOK_KEY --host desk.acme.example --signing`); the rule names it, never its value. A rule may name only such a key: another secret of the tenant (a password, a token, a signing key for another host) is refused when the rule is set, and the edge refuses to sign with it at every delivery, so whoever may manage notifications learns nothing of a secret from a signature. The refusal when the rule is set comes only to someone who may read the tenant's secrets (`secrets.read`), and so does the approval of a rule that waits for another admin: anyone else gets the same answer whether the key is set or not, a warning that says what the key must be, and the edge's check at every delivery still holds. What the rule's deliveries say afterwards is just as neutral: a delivery whose key cannot be used records one error, "nothing is signed with tenant secret NAME for HOST", whether the key is missing, is not a signing key or is bound to another host, and is tried again like any other failure; the edge's log says which it was, and someone with `secrets.read` sees the key and its hosts with `oz0 secret list`. A signing key is never sent to any host itself. A delivery that fails is tried again at 30 seconds, 1, 2, 4 and 8 minutes, six attempts in all, each kept on the delivery (`oz0 notify rule list`). A `4xx` other than `408` and `429` is final. Redirects are not followed. Webhooks go to public addresses. A private, loopback or link-local address is refused unless the operator allows its network (`orchestrator-zero server start --notify-allow-net 10.20.0.0/16`), which is how a webhook to your own intranet works. Deliveries share the edge's connections: at most eight at once to one receiver, two kept open between deliveries, each closed after 30 seconds idle. A receiver that holds its connections open costs the edge no more than that. Use `https://`. Over plain `http://` the request, its signature and any arguments travel unencrypted, so the edges post to an `http://` URL only on a network the operator allows (or on a dev server). Putting a rule with an `http://` URL still works, and the CLI and the console warn about it. ### Verify a delivery Check the signature before you read anything in the body, compare it in constant time, and refuse a timestamp more than five minutes from your clock: ```python [receiver.py] import hashlib, hmac, time def verify(secret: bytes, headers, body: bytes) -> bool: ts = headers.get("X-Oz0-Timestamp", "") if not ts.isdigit() or abs(time.time() - int(ts)) > 300: return False want = "sha256=" + hmac.new(secret, ts.encode() + b"." + body, hashlib.sha256).hexdigest() return hmac.compare_digest(want, headers.get("X-Oz0-Signature", "")) ``` Keep the `Idempotency-Key`s you acted on for at least the replay window, and skip one you saw. ### Decide from your system: submit and poll A webhook tells your system that an approval waits; it never carries a way to decide. Your system decides like any other caller, through the [runtime API](https://docs.oz0.ai/build/apps/runtime-api) with an [API key](https://docs.oz0.ai/admin/tenants) whose role may decide (operator or above), and then polls for the outcome. A key decides only the approvals it is named for, so the webhook target names the key its receiver decides with: `--decide-key` takes the key's ID (the 12 hex digits after `oz0_`, as `oz0 apikey list` shows them). That key may then decide the approvals this target was sent (a delivery that is waiting, was sent, or failed and is tried again; not one the edge gave up on or refused, which may never have reached the receiver), for as long as the target names it and goes to the same address; any other key is refused with `403`, and so is this one for approvals the target was not sent, unless the approval's [deciders](https://docs.oz0.ai/agents/approvals/configure#who-decides) name it. Naming a key, sending it other approvals or moving its webhook is a [reviewed change](https://docs.oz0.ai/govern/reviewed-changes) (`notify_decide_keys`): `oz0 notify rule put` takes `--reason` then, and with four eyes on it waits for another admin. ```bash [Terminal] curl -sS https://edge.acme.example:7443/oz0.runtime.v1.RuntimeService/DecideApproval \ -H "Authorization: Bearer $OZ0_API_KEY" -H 'Content-Type: application/json' \ -d '{"job": "job-1e71252a-...", "id": "a-5ccd30628bdc", "approve": true, "comment": "the desk approves"}' curl -sS https://edge.acme.example:7443/oz0.runtime.v1.RuntimeService/ListApprovals \ -H "Authorization: Bearer $OZ0_API_KEY" -H 'Content-Type: application/json' -d '{"job": "job-1e71252a-..."}' ``` The decision is recorded as the key's, under the same rules as a person's: the approval's [deciders](https://docs.oz0.ai/agents/approvals), the content permission for an edit, and the audit trail, whose `run.approval.decide` entry says which rule named the key (`key_named_by`). ## Chat channels A rule may post to a [chat channel](https://docs.oz0.ai/build/apps/chat-channels) of the tenant: Microsoft Teams, Slack or Discord. Two kinds of target, which one rule may mix: - A room, `--channel :`, such as `--channel "teams:19:ops@thread.tacv2"`. For Teams the room must be one the bot has heard from (mention it there once), since Teams names where replies go in each message. The edge keeps the room while rules post to it or people mention the bot there, and forgets it after 30 days of neither; mention the bot there again then.
The room's readers are people the platform does not know, so the post carries the event's facts only: the agent, the function, the level, the deadline and the link to the job, never the call's arguments, its marked fields' values, or a question's words. - People in their personal chats with the bot, `--channel-person teams:ana`, `--channel-team teams:payments`, `--channel-role teams:admin`: each current member of the tenant it names who [linked their chat identity](https://docs.oz0.ai/build/apps/chat-channels#link-your-own-chat-identity) gets it in their one-to-one chat with the bot: the Teams personal chat, the Slack direct message to the app or the Discord DM with the bot they last wrote in.
A slash command never names that chat, even one that links them, since the room it is typed in is one others read: someone who linked with a command is reached once they write to the bot directly.
A person who holds the content permission and may read the job's chat sees the call's details on the card (and Edit and approve); anyone else gets the facts. Someone with no such chat is left out of the delivery and counted, never named. ```bash [Terminal] oz0 notify rule put approvals-to-teams --event approval.waiting --event job.failed --agent support-triage \ --channel "teams:19:ops@thread.tacv2" --channel-team teams:support-leads ``` A waiting approval or question comes as the channel's card, with its buttons, and is decided from it as from any thread: by the clicker's mapped account, which must be allowed to decide it (the operator role, the approval's [deciders](https://docs.oz0.ai/agents/approvals), never the starter under four eyes), and only on the card the platform posted. A job that failed or a budget's share comes as a message. Each delivery posts once however often the notifier tries it; a post the chat system rate-limits, or that no node of the plugin took, is made again after its wait, as a thread's posts are. ## What is counted A notification names who was notified in its delivery rows and counts people by kind in the audit trail (`run.approval.escalated` says "3 people who may decide", never who). Nothing counts or lists notifications or decisions per person. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.