Call bindings and budget charges

For: cluster operators and plugin authors.

Calls bound to the node

Every activity task the edge hands a node carries a call binding: a token, MACed by the edge, that names the job's envelope, the workflow, its run, the activity and its attempt, the function of a tool call and the digest of its arguments, the node and its certificate, and when it expires. The node's runtime sends it with every model call, every connection token request and every harness session's call of that activity. The edge refuses (403 oz0_binding, audited run.binding_refused) a call whose binding:

• comes from another node, or another certificate of the same node;
• names another job than the call does, or another function or plugin than the token asked for;
• was changed, forged or has expired.

So a token or a binding lifted from a node, or from a sandboxed tool on it, is useless anywhere else, and a node can no longer meter its calls to a job it does not run. A colleague's answer is metered to the job that asked, in the same tree, and that is allowed.

server start --call-binding says what happens to a node's call that names a job, or asks for a connection token, and carries no binding:

A model call that names no job would meet no job's budget, quota, label rule or data: local, so one with no binding is refused too, under required and from any node whose runtimes carry bindings, in every mode.

The runtime names its job on every model call it makes, and a harness session's and a tool's keys name theirs, so no call of a current node is touched.

Three kinds of caller still make such a call, each on purpose: an operator, whose call through the gateway is no node's; the platform's own calls, such as a model canary's; and a node the edge still takes unbound calls from, an older runtime in when-present or auto, whose call is metered to no job and meets the tenant's own budget and data rules alone.

In auto, the edge reads which runtimes carry bindings from the nodes' heartbeats, every few seconds: each runtime reports the features its bundle names, and a runtime that carries bindings says call-binding.

A node counts when it runs (or drains) a runtime without that feature, or its supervisor is too old to report features, whatever its runtime; it counts from its heartbeats of the last ten minutes, so a node that was connected before a server restarted counts before it reconnects, and a node that has just connected counts until its first heartbeat shows its runtime.

Until the edge has read the heartbeats once after it starts, it lets no such call through. What one node reports answers for that node alone: every other node's calls still need their binding.

What a node reports is the node's word, so it only ever loosens the check once. As soon as a node's heartbeat shows that every runtime it runs carries bindings, the edge that holds its control stream marks the node in its stored status, which every edge reads; from then on no edge takes that node's calls without a binding, whatever its later heartbeats say, until management tells the node to run another runtime or binary (oz0 node upgrade, or a rollout or its rollback).

The edge logs a node that reports an older runtime after it was marked. Even for a node that counts, a call without a binding acts only as the when-present row says.

Once every node has upgraded, pin --call-binding required, which takes nothing on a node's word; the edge logs when the answer changes.

A child's admission is asked from the parent's workflow task, which no binding covers, so in every mode the edge takes it without one only from the node it handed the parent's latest workflow task.

What the audit log records

Audit entries name the envelope and the binding of what they record, with the level fields and the action class of the function, as fields the entry's hash covers (version 3 of the audit trail's entries).

Origins and sequences

Before each tool call is scheduled, the edge writes run.tool_intent with the call's receipt: every sensitive argument's role, floor and origin (request, identity, record, any or none) as unit digests, and the sequence rules that decided it.

On an edge that reaches the database through management, the entry is written by management or kept on the edge's disk first.

run.tool_call and run.child_job record what each result gave the job's origin index, where in the history the step was, and the digests of each call's sensitive arguments as it was made; run.origins records each job's start (a child's lists the entries it inherited from its parent's index). Every digest in the trail is keyed per tenant.

The intent entry also carries the floors the tenant's policy sets for the call's sensitive arguments, as the edge reads them from its own copy of the policy and the plugin's approved listing: a receipt is the node's word, and the verifier holds each verdict to the edge's floor, never only to the one the receipt names.

The verifier finds each call the history holds by what the edge saw itself, its run, its activity and the digest of its input, never by a name or a call ID the node wrote, and the receipt the call's input carried must be the one the edge recorded.

Every argument the call was made with that the edge set a floor for needs its verdict, and a verdict about an argument the call was not made with is a difference.

A call the edge set a floor for that carries no receipt is held to that record as well, whatever the job's own records say: when the history shows it made with such an argument, or holds a call at that activity it cannot match to its input or its arguments, it is a difference (a shadow run's read-only calls, which keep no index, are the exception); when the exported segment holds no record of the call at all, the report says so in a note.

A run never schedules two activities under one ID; a history that shows it is a difference too.

A job keeps the policy its tree started with, so a receipt may name a weaker floor than the edge's only where a tool policy in force since the job's tree started set no stricter one: the verifier reads those policies from the segment's tenant.tool_policy entries with the function's marking the intent entry records, counts such verdicts apart (tightened) and decides them again against the floor the receipt names.

When the segment holds neither the tree's start nor the policy in force then, nothing weaker is accepted: export the segment from before the job started.

oz0 evidence verify rebuilds every index from the trail, decides each verdict again, compares each receipt with the arguments and with the edge's floors, and prints the differences (0 expected) and the share of calls with a sensitive argument whose origin was checked against a floor. See tool policy.

Budget charges

Every model call a budget applies to reserves its worst case before it is sent and settles on the reply (Each call reserves its worst case). The gateway's own record of each call carries what the call reserved, how it settled (metered for a complete answer, unsent for a call whose request never reached the provider whole, unknown otherwise), what it was charged and the kinds of budget it held (tree, tenant, quota; never whose quota), and, for a node's call, the attempt the edge handed out with the call's binding. oz0 evidence verify then checks, without the cluster:

• that each charge follows from its record: what was metered for a complete answer, at least the reservation for anything else (budget_charge_mismatch otherwise, as is a call that broke off but was charged only what was metered, or one settled as unsent that the provider answered);
• that each reservation's key is its attempt's (reservation_key_mismatch);
• that the attempt a node's runtime named is the one the edge handed out (attempt_unverified), for every attempt of a retried model step: a runtime cannot hide a call of its last attempt under an earlier attempt's number. A record without a binding (an operator's call, a node under when-present) is counted, not trusted.

It reports how many calls reserved, what they were charged and what was metered of them, and how many were of unknown outcome. A budget passes its limit by at most one call's reservation (none with reserve: strict). See budgets for the reservation rules.