--- title: "Check a plugin candidate" description: "Test a candidate against a tenant before installing it." canonical_url: "https://docs.oz0.ai/build/plugins/develop-and-test/candidate-checks" --- # Check a plugin candidate > Test a candidate against a tenant before installing it. For: plugin authors. Test a candidate against a tenant before installing it. ## Try it on a real cluster The dev server has none of your cluster's secrets, plugins, models or networks. To try a working copy there without touching anyone, make it a private candidate on nodes you choose: ```bash [Terminal] export OZ0_CONTEXT=~/.config/orchestrator-zero/prod/context.json oz0 dev ./my-plugin --server https://oz0.example.com:8443 --tenant default --select pool=dev ``` ```text Private candidate 4f1c09a2be7d: my-plugin 0.4.0 on 1 node dev-box-1 in tenant default (they leave my-plugin 0.3.2 meanwhile), owned by user:alice. Only jobs that name it reach it: oz0 run --candidate 4f1c09a2be7d --tenant default my-plugin 'your input' my-plugin 9a1e0c3f77b2 ready on 1 node in 9.8s: my_plugin.hello Watching for changes; Ctrl+C stops and removes the candidate. ``` Nothing starts on your machine. The working copy is packed and sent to the cluster: - Its nodes: the tenant's online nodes with the `--select` labels (every node without it) run it instead of the installed version, next to the tenant's installed plugins, with the secrets the installed version gets. ### Its jobs The candidate answers only on its own queues, `cand..`, one for each pool its agents and flows fall in: with [shared pools](https://docs.oz0.ai/concepts/routing#shared-pools) (the default) the tenant's pool of their node selector and labels, such as `cand.4f1c09a2be7d.pool.~342aaaea`, which carries only the candidate's jobs. Only jobs started against it reach it: `run --candidate `, or `candidate` in the runtime API's `StartJob`. The tenant's other jobs, webhooks, routines and events never do, and child jobs of its own plugin stay on it. Which plugins are the candidate's, the node learns from the cluster with the candidate, never from a queue's name. - Nothing else on its nodes: while it runs there, its nodes take no other work of the tenant: no other plugin's jobs, no `oz0.default` tool calls, no events, no models served to the fleet and no queues of their `queues` label. No new job makes them its home. ### Jobs already homed there A job keeps the node where it started as its home, and makes its tool calls, hooks and other calls there. A job that started on one of the nodes before the candidate arrived (an agent waiting on an approval or an event, say) would meet the candidate's code, so the node refuses its calls instead while the candidate runs. Nothing of a refused call ran, so the job waits and makes it again (after 30 seconds, then a minute, two, four, eight and sixteen minutes, then every half hour), and the node answers it with the installed version once the candidate leaves. A call still refused after a day, or once the job's history has 10,000 events (many calls waiting at once), fails with `CandidateNode`, whose message says to retry the job; a job cancelled meanwhile is cancelled. `dev` warns when a node it takes is the home of running jobs. Select nodes of a dev pool, which run nobody's jobs, and nothing of anyone's is touched. ### Apart from the node's other plugins On a node with sandbox tier 1 (bubblewrap on Linux, Seatbelt on macOS; `oz0-node doctor` says), its tool servers run in the sandbox: they read their own plugin folder and write a folder of their own, see none of the node's other processes, files or credentials (so none of the other plugins' secrets), get their own secrets from the node, and reach only the hosts the tenant's and the plugin's egress lists allow. Its harness sessions run there as on any tier-1 node. Its environment is built without running its code: wheels only, the node's own Python, and its own Python project with the `uv_build` backend that `oz0 plugin new` writes (`requires = ["uv_build>=0.9,<0.13"]`, `build-backend = "uv_build"`, nothing else), whose versions take in the node's uv (uv then builds it itself), or none. A project that builds with setuptools or hatchling, sets `backend-path`, asks for a `uv_build` range without the node's uv or takes `uv-build` from a source of its own, is a uv workspace (in `pyproject.toml` or `uv.lock`), takes a dependency from a folder or a Git repository, has a `uv.lock` that marks it editable without a build system, or sets a `[tool.uv]` setting about Python or builds cannot be tried as a candidate; the node says which. The node also leaves out the bundle's `.venv`, `.python-version`, `uv.toml` and `.npmrc`, and builds its own. A node without tier 1 runs it as the node's user, next to the other plugins, and `dev` warns. The node's installed plugins are not isolated from each other. - Reloads: every save reloads it on the same queues. - The end: Ctrl+C removes it, and its nodes go back to the installed version. If your machine goes away, the server removes it once dev stops renewing it, 90 seconds after the last renewal (`--candidate-ttl`, 10s to 10m). - Its ID is the same each time you start `dev` for the plugin in the tenant, so a restart reuses its queues. A new `dev` takes the candidate over from an old one, which then stops. The cluster refuses a candidate: - that would leave a plugin the tenant has installed with no other online node, since its nodes leave every pool meanwhile; - that would take the last online node that serves a local model to the fleet, or the last that polls a queue of its `queues` label; - on a node that already runs a candidate, or that tests a plugin's quality gate; - while a rollout of the plugin runs. A gate or a rollout that later reaches one of its nodes wins there: `dev` says when no node runs the candidate, and `--candidate` jobs are refused until one does. ### Candidate permissions A candidate requires the tenant's operator role and content permission because its code can read the content of the jobs it serves. Candidate code runs on the selected nodes; without sandbox tier 1, it can also see what those nodes hold. #### Eligible nodes and secrets An operator can use only nodes an admin opened with `candidates=allow`, normally set on the join token. An admin can use any node. An operator's candidate receives only secrets available to the plugin's installed version. It receives none if the plugin is not installed. A working copy requesting more is refused, although an admin may try it. If an admin removes a secret from the installed plugin or uninstalls the plugin, its running candidate loses that secret too. #### Secret bindings and network access An operator's candidate must receive each secret in the same way as the installed version. The platform refuses a working copy that: - Adds, removes or changes a host binding in `requires.inject`, or delivers secrets differently through `requires.secrets_from`. - Adds a `secret_arguments` mark that the installed version lacks. - Adds or repoints an outside agent whose `harness.auth` receives a secret. - Adds or changes its own outside harness. - Adds or changes its own remote harness (`protocol: remote`) while holding a secret. - Adds a reachable host while its code receives a secret value. This includes adding an `egress` entry, removing a deny entry, or changing a remote tool's address or headers. Network access is compared for each process. A tool receives the plugin's `egress`; an agent sandbox receives the plugin's list combined with its own. Moving a host from an agent's list to the plugin's therefore counts as a change. Removing a binding would give code the value itself, which it could send to any permitted host. A secret counts as reaching code as a value in any of these cases: - It has no binding. - An outside agent names it in its auth. - The plugin has its own remote harness, whose sessions receive the secrets unchanged for the provider. - A tenant binds it to a marked tool argument, so the call receives the filled value. On Windows or a node without sandbox tier 1, every secret counts as a value because a binding does not hide it from code. A candidate holding a secret on such a node may add no host. See [where injection applies](https://docs.oz0.ai/admin/credential-injection#where-it-applies). Refusals identify the change, never the secret value; an admin may try the change. #### Connection definitions An operator's candidate must keep the installed version's connections unchanged. The edge derives a call token from the connection definition: its tenant secret, tools, functions, scopes, resources and every `token` field. Adding, removing, renaming or changing a connection is refused. Reordering keys or replacing an absent list with an empty one is not a change. If the plugin is not installed, an operator's candidate can declare no connections. An admin may try a changed definition, and the refusal names the connection without exposing a value. No candidate, including an admin's, may take another plugin's connection. As at installation, a connection name declared by another installed plugin, or a secret named by another plugin's connection, is refused. #### Tool locations and declared grants An operator's candidate must run each tool where the installed version does. The runtime acts as a remote tool's client, keeping call content out of plugin code. The platform therefore refuses a change from remote to local or local to remote, a move to another connection or off a connection, or an address change for a connected tool. An admin may try those changes. The candidate must also retain the installed declarations under `uses:`, `reads:`, `mail:` and `requires.models`, and each tool's `uses_model`. Grants belong to the plugin name, so adding an entry, mail connection, sampling alias or gateway format is refused for an operator. An admin may try it. While a plugin's first version waits at its quality gate, or after that gate stops, it has grants but no installed version. An operator's candidate may then declare none of these permissions. A plugin with no tenant record has no grants, so an operator may try a new one that requests them. #### Enforcement when the edge reads the registry The edge rechecks the candidate whenever it reads the registry. Where injection applies, a host-bound secret reaches an operator's candidate as a placeholder. The node proxy fills it only for the installed version's hosts, regardless of the candidate manifest. A tenant argument binding fills only an argument also marked by the installed version. It fills none when the plugin is not installed. A candidate whose own remote harness differs from the installed version receives none of the installed version's host-bound secrets. Calls receive connection tokens according to the installed definition. They receive none for a connection absent from that version or for a tool running somewhere different. Tenant bindings of installed connection tokens apply to the candidate too. An admin's candidate uses its own connection definitions. The edge does not recheck outside agents, `uses:`, `reads:`, `mail:` or `requires.models`. Management's save checks and each renewal prevent additions, repointing or wider declarations. See [remaining injection limits](https://docs.oz0.ai/admin/credential-injection#not-yet). #### Renewals and changes of owner permissions Permissions are decided for the person saving the candidate. If that person is not a tenant admin when renewing or resending the same bytes, the platform repeats the save checks against the current installed version and nodes. It checks content permission, secrets and their delivery, connections, tools, grants, and whether the nodes remain open to candidates. New bytes are a new save. A reload keeps the same nodes, so those nodes must still be open. A refused reload leaves the current code running until the next renewal ends it. A comparison renews its candidate on behalf of the person who started the comparison, regardless of who saved the candidate. Whenever that person is not a tenant admin, the same checks apply. If renewal ends the candidate, the comparison stops at its next pair of runs. A candidate placed through an API key depends on that key. Revocation or expiry removes its permissions regardless of the creator's permissions, and the next renewal ends it. A candidate saved by an admin that still passes the operator checks continues as an operator's candidate, with no change for its jobs. One that fails those checks is refused and ends; `dev` explains why and stops. Continuing with the installed version's secrets and connections would test neither the original candidate setup nor the installed version accurately. A renewal by an admin changes nothing. Admins may remove anyone's candidate. Every step is in the audit log: `candidate.create`, `candidate.update`, `candidate.run`, `candidate.standing` (one an admin saved now counts as an operator's), `candidate.remove` and `candidate.expire`. The Plugins page shows the tenant's candidates with their owners and nodes, and `plugin candidates` lists them. `--server` is the cluster's management URL, which your CLI context must be for, or the context file itself. On a real cluster the candidate installs no dependencies: `dev` warns about the plugins its `uses` names that the tenant has not installed. ### A changeset's candidate check The platform places a private candidate of its own for an agent-made changeset to an installed plugin, before the changeset's changes approval, when the job's plugin holds a [results grant](https://docs.oz0.ai/improve/evidence-for-agents#the-grant) for the plugin it changes: the [Plugin improver](https://docs.oz0.ai/build/plugins/verified/plugin-improver/job-lifecycle#the-candidate-check)'s changesets, for one. Such a candidate is built from the changeset alone (the job's base archive plus its patch, checked file by file by the edge), is never placed when it widens what the plugin may reach (a secret's binding to hosts dropped or changed counts), goes only on nodes labelled `candidates=allow` that have sandbox tier 1 and are not the job's own node, gets only the secrets the installed version gets, is owned by the job (`job:`), and leaves when its [comparisons](https://docs.oz0.ai/improve/comparisons-of-a-changed-agent#a-candidate-check) end, or by its time to live if the server stops. It is never installed or promoted. Open a node to these candidates as to an operator's, with `candidates=allow` on its join token; without such a tier-1 node the approval says the check did not run. ## Preview an install against a real tenant ```bash [Terminal] oz0 plugin preview ./my-plugin --tenant team-a oz0 plugin preview ./my-plugin --tenant team-a --json # for the plugin's CI ``` `preview` packs and uploads the working copy as `plugin install` does, runs the install's checks against the tenant and keeps nothing: the names, the tool IDs and what the install would refuse (a tool ID or an agent name another plugin of the tenant has, the tenant's queue limits, contracts that break the installed version's without a new major version), the contract changes, and what the tenant's plugins would take of its limits. It exits non-zero when the install would be refused. ```text my-plugin 1.2.0 from bundle /home/you/my-plugin, would lock to sha256:5f0c... Tools: my_plugin Agents: my-plugin The tenant's plugins would have 7 of 1000 agents and flows, on 4 of 50 task queues. error: the install is refused: agent "my-plugin" is already defined by plugin other in tenant team-a error: an install would be refused: 1 errors ``` ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.