Sandbox egress rules
For: cluster operators and plugin authors.
The egress proxy
Host lists and address checks
The egress proxy allows a request when the host is on the list of the tenant, the process's plugin or its agent, and no deny entry names it; it denies the rest. It resolves names itself and checks every address it is about to connect to, also for an entry that names an address.
An internal address is refused unless an entry names that address. This includes:
- Loopback (
127.0.0.0/8),0.0.0.0/8, multicast, reserved ranges and the node's current interface addresses. - Link-local addresses, including metadata at
169.254.169.254and Scaleway's169.254.42.42, plus Oracle Cloud's192.0.0.0/24. - Azure WireServer
168.63.129.16and Alibaba Cloud100.100.100.200. - IPv6 metadata: AWS
fd00:ec2::/32(IMDS and the EKS Pod Identity agent), Akamaifd00:a9fe:a9fe::1, Scalewayfd00:42::42, GCPfd20:ce::254, Oracle Cloudfd00:c1::a9fe:a9feand Alibaba Cloudfd00:100::100:200. - IPv4-hidden addresses: NAT64 (
64:ff9b::/96) when it contains an internal address, and 6to4 (2002::/16), Teredo (2001::/32) and local-use NAT64 (64:ff9b:1::/48) whatever address they contain.
Only an entry of the tenant's list may name an internal address. A name that resolves to no address is not connected to.
Private networks
An entry of the tenant's list reaches a private address (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, carrier-grade NAT's 100.64.0.0/10, IPv6 unique local fc00::/7 and the deprecated site-local fec0::/10, and their IPv4-mapped and NAT64 forms) by name or by address, so a sandbox can reach a company's own services when a tenant admin lists them; list them only when it should.
An entry of a plugin, an agent or a harness reaches public addresses only: a name it allows that resolves into a private network is refused when the connection is about to be made, and so is a private address it names.
The ranges that are not on the public internet although they are neither internal nor private (documentation, benchmarking's 198.18.0.0/15, discard and the IETF protocol assignments) are treated as private networks.
The node's Go code and its Python runtime and plugins hold one table of these ranges, which their tests check against each other. Otherwise a plugin's author, or whoever runs the DNS of a name a plugin lists, could point the sandbox at the company's network.
Where an entry of the tenant's and another's both allow a target, the tenant's decides.
A client may also ask for public addresses only, for one request, with the header Oz0-Egress-Public-Only: 1, which the proxy does not forward: the tenant's names are then held to public addresses too, and only a private address the tenant's list names itself is reached. oz0_web.fetch sends it on every request.
TLS tunnels and server names
Inside a CONNECT tunnel the proxy reads the TLS ClientHello before anything reaches the server, without decrypting it, and requires its server name (SNI) to be the host the tunnel was opened to: a client cannot open a tunnel to an allowed CDN and name another site in the ClientHello (domain fronting by server name). A tunnel to a name must send a server name; one to an address the list names may send none.
A tunnel that does not start with TLS is refused unless its entry ends in /tcp, and so is one that sends nothing for 10 seconds: a protocol in which the server speaks first (SMTP, MySQL, FTP) needs a /tcp entry.
Raw TCP entries
Under a /tcp entry the server may speak first, a tunnel that starts with a ClientHello still has its server name checked, and one that begins as a plain HTTP request has that request's Host header and target held to the tunnel's host, and is refused and audited as a plain request is.
Bytes go on as they come until a line shows itself a request, so a protocol whose client speaks first, with or without a line end, gets its answer, unless its line reads as a request that names another host: a CONNECT line is one only when its target reads as a host, so NATS and Go's net/rpc over HTTP pass; a request is held from the end of its first line until its head ends, and refused if that takes more than 10 seconds or 64 KiB.
A line with no version may be a command that waits for its answer, so it goes on, and the lines after it are read as they pass: a Host header among them holds the head the same way.
Those first bytes are all the proxy reads: after them a /tcp tunnel is an opaque pipe, so a later request on the same tunnel, HTTP/2 in clear text (whose host travels in compressed frames), TLS that starts later in the stream (STARTTLS) and whatever any other protocol carries go unread, and neither the Host check nor the secret scan sees them.
A /tcp entry for a shared host is a way to all of that host's customers: give one only to a host that is yours or serves one customer.
Shared hosts and encrypted request limits
What the proxy cannot see without decrypting is the HTTP Host header inside TLS: a client that names the allowed front in the ClientHello and another customer of the same CDN in the encrypted Host header reaches that customer on a CDN that does not require the two to agree. Package registries and other popular hosts often sit behind such CDNs.
This stays open on purpose. Closing it for every host would put a certificate authority the proxy holds into every sandbox, and break every client that pins its certificates; the proxy decrypts only for a host a credential is bound to, where the two must agree.
So allow shared CDN names only when you must, and for a package host behind one, put a mirror of your own on the tenant's list instead and leave the shared name off.
With Encrypted Client Hello the visible name is the provider's public one, which does not match, so such a tunnel is refused; GREASE ECH, which browsers and recent TLS libraries send, carries the real name and passes.
What the check holds is the visible name: a client that carries an ECH configuration of its own for a provider whose public name is the allowed host itself sends that name, and its tunnel passes with the real name hidden, so, as with the encrypted Host header, a tunnel to an allowed host reaches that host's provider, not only that host.
A plain HTTP request whose Host header names another host than its URL is refused.
An address with an IPv6 zone ([fe80::1%eth0]) is neither an entry nor a target, since a zone would hide the address from these checks.
Requests without the proxy's token are refused on macOS, where the proxy listens on the node's loopback; on Linux it is a socket only the sandbox reaches.
The proxy sees what a sandboxed process connects to. What a provider fetches for a model call never passes it, so the gateway holds those addresses to the same list.
Denials and troubleshooting
A refused request gets 403 with a body that names the host and says how to ask for it (or, inside a TLS tunnel, TLS's access_denied alert). It is logged on the node, streams to the job's live view with its reason, shows on the job (job get, the job's page, which proposes the entries that would allow it) and goes into the audit trail as egress.denied, each target once and at most 20 per job.
A reason can quote what the process sent, a Host header's value for one, so on the job, in the live view and in the answer to a start or a routine's fire that finds its job already there, a reader without the content permission gets the host, port and kind without it, and the audit log keeps it in the entry's private part.
When a name was refused for what it resolved to, the body names the kind of address (loopback, link-local, a private network) and never the address itself: the sandbox has no resolver of its own, and a refusal must not become one.
The node's log and the job's denial record keep the address, for the operator and oz0 egress learn.
Learn from a job's denials
oz0 egress learn <job> turns a run's denials into the narrowest entries that would allow them, with the oz0 tenant egress command that sets the tenant's list; it applies nothing. Every entry it proposes is a valid allowlist entry, and every word of the commands it prints, and of the command in a refusal's body, is quoted for a POSIX shell, so pasting one runs exactly the words shown.
A request whose target is not a host name or an address is malformed (400), not a denial: it is never recorded or proposed, and a recorded target that is none is listed for review, escaped, with no entry.
Refusals that no entry should change (a mismatched server name or Host header, a deny entry) are listed for review.
An internal address, such as a probe of the metadata service at 169.254.169.254, is refused as internal and never proposed as an ordinary entry: learning shows it for the tenant's list only, with a warning, and leaves it out of the agent's YAML and the printed command.
A private network refused to a plugin's or an agent's entry is proposed for the tenant's list only, and is in the printed command.
A request to an allowed host whose connection fails gets 502 and is logged on the node, but is not a denial: no allowlist change would help.
Widening any of the lists, the tenant's or a plugin's at install, is a reviewed change that waits for a tenant admin, with what each new entry widens (Reviewed changes).
| List | Set with |
|---|---|
| The tenant's | oz0 tenant egress <tenant> api.github.com '*.pypi.org', or Govern › Policy › Egress |
| A plugin's | egress: in oz0-plugin.yaml |
| An agent's | egress: in its definition |
| Entry | Allows |
|---|---|
api.github.com | the host, on ports 80 and 443 |
*.example.com | every subdomain, at any depth, not the domain itself |
jenkins.*.ci.example.com | a * label elsewhere matches exactly one label; the last two labels are names |
pypi.org:8443, registry.local:* | the host on that port, or on any port |
10.0.0.7:5000, [2001:db8::1]:443 | an address; a private or internal one only in the tenant's list |
github.com:22/tcp | raw TCP to it: a tunnel that need not carry TLS (the port is required) |
!uploads.example.com | a deny entry: never this host, on any port unless one is given, whatever another entry or list allows |
There is no allow-all entry, and a * is a whole label. An internal address, or one on a private network, can only be on the tenant's list: plugin lint refuses it in a plugin or agent, nodes drop it, and the proxy would refuse it anyway.
Review download hosts
A host that serves files anyone can publish (a code host's raw files and archives, a package registry, an object store, a paste site or a tunnel) is a download path: whatever anyone puts there, a session that may reach it may fetch and run, which is all a model with a backdoor in its weights needs.
The proxy cannot tell a package from a payload on such a host. So wherever a list is widened to one, the review says so with that reason: the change request's finding, a plugin configuration's egress line, oz0 egress learn, the install screen, the console before saving, and plugin lint (OZ-S022).
It never refuses and never changes who approves: a registry or a plugin's own download host is often what a tenant needs, so the note is a reason to look and the admin decides (Reviewed changes). Prefer the narrowest host (files.pythonhosted.org, not *.pythonhosted.org), or a company mirror on the tenant's list.
Verified plugin tools
The same lists bound the verified plugins' tools that reach hosts for a model, on every node, sandboxed or not: oz0_web.fetch, net-diag's probes and cve-triage's OSV lookups check each target against the tenant's and their plugin's list first, connect to the address they checked, refuse a name that leads to the node itself, its link or a metadata service, and hold a target only their plugin's list allows to public addresses, as the proxy does; oz0_web.fetch also refuses any private address that only a name leads to, the tenant's names included.
In a job's sandboxed tool server they match names against the lists without resolving them and go through the egress proxy by name, which holds them to the same rules; oz0_web.fetch asks it for public addresses only (Oz0-Egress-Public-Only), so it never reaches a private network by name there either.
Every tool process learns the lists (OZ0_EGRESS_FILE, OZ0_PLUGIN_EGRESS) and, on a tier-1 node, how to run a command of its own in the sandbox, which pr-reviewer's test runner does for a pull request's tests, each in a sandbox of their own.
A sandbox cannot start another, so a tool that does this declares sandbox: own: in a job an outside event started, its calls run in its plugin's own server on the job's tier-1 node instead of the job's sandbox, which is allowed only for a plugin that holds no secret, setting, connection or egress, only for a call the job's workflow marked (never a hook's or a harness session's), and each such call is audited as tool.own_sandbox.
Such a server still runs as the node's user, so pr-reviewer's unpacks and reads a pull request only in a sandbox it starts. Other plugins' tools may honour them; only the sandbox enforces them.
Changes and backend limits
Removing a host from an egress list refuses new connections. An already-open tunnel stays open until it closes. A container or microVM started with no egress has no network to enable; restart it after adding entries. A remote sandbox takes its list when it starts.
The remote service applies a remote sandbox's combined list, without distinguishing tenant entries from plugin entries. At tier 0, a third-party tool's own checks are the only enforcement. See remote sandbox egress.
For an allowed host without a credential binding, a connection failure's 502 can include the dialer's error and the address. A bound host's failure uses a fixed reason.