--- title: "Deploy with Kubernetes" description: "Prepare the chart and node isolation, then install the cluster." canonical_url: "https://docs.oz0.ai/fleet/kubernetes" --- # Deploy with Kubernetes > Prepare the chart and node isolation, then install the cluster. For: cluster operators. The chart in `deploy/helm/orchestrator-zero` runs a [cluster](https://docs.oz0.ai/fleet/cluster) on Kubernetes: two or more servers on a PostgreSQL you provide, each running management, the edge and Temporal's services. Nodes can run in the cluster too, or join from outside it. ## Choose a task
Task Guide
Helm objects and values Open guide
Maintain a Kubernetes deployment Open guide
## What you need - Kubernetes 1.29 or later and Helm (the chart is tested with Helm 4.3). - PostgreSQL 13 to 16, reachable from the pods, with `max_connections` of at least 100 per server. The user in the URL should be allowed to create databases; otherwise create `oz0`, `oz0_temporal` and `oz0_temporal_visibility` yourself, named after the database in the URL. - The images, pushed to a registry your cluster pulls from. No official images are published yet; build them with `make images` (below). - A network plugin that enforces NetworkPolicy, if you want the chart's policies to mean anything. Most do; kind's does. ## Build the images ```bash [Terminal] make images IMAGE_TAG=0.9.0 # orchestrator-zero:0.9.0 and orchestrator-zero-node:0.9.0, built locally docker tag orchestrator-zero:0.9.0 registry.example.com/oz0/orchestrator-zero:0.9.0 docker push registry.example.com/oz0/orchestrator-zero:0.9.0 # and the same for orchestrator-zero-node ``` `make images` never pushes. Both images are based on `debian:trixie-slim` pinned by digest and run as user 65532 on a read-only root file system.
Image Holds
orchestrator-zero The server binary, git for plugin installs from Git, and the dist/ folder the edge serves to new nodes: node binaries for Linux, macOS and Windows and the runtime bundle, signed with the build machine's release key ( ARCH=arm64 builds for arm64; PLATFORMS=linux/amd64 limits the node binaries it serves).
orchestrator-zero-node The node binary, with uv, Python and the runtime already installed by oz0-node prepare , and Debian's bubblewrap 0.12.0 for sandbox tier 1 . A node starts in seconds without reaching the internet.
The node image works outside Kubernetes too: ```bash [Terminal] docker run -d -e OZ0_SERVER=https://edge.example.com:7443 -e OZ0_JOIN_TOKEN='K10...' \ -v oz0-node:/var/lib/orchestrator-zero-node orchestrator-zero-node:0.9.0 ``` Its entrypoint copies uv, Python and the runtime into the volume unless they are there already, joins once when the volume holds no identity (with `OZ0_JOIN_TOKEN`, or `OZ0_CA_HASH` to wait for an operator's `node accept`), and runs the node. Keep the volume at `/var/lib/orchestrator-zero-node`: the runtime's virtual environment refers to its Python by that path. ## Make the master key and the Secrets The master key unseals the cluster's CAs and secrets. Make it yourself, keep a copy outside the cluster, and give it to the chart as a Secret, with the database URL in another: ```bash [Terminal] kubectl create namespace oz0 head -c 32 /dev/urandom | base64 > master.key kubectl -n oz0 create secret generic oz0-master-key --from-file=master.key kubectl -n oz0 create secret generic oz0-db \ --from-literal=url='postgres://oz0:@db.internal:5432/oz0?sslmode=require' kubectl -n oz0 create secret generic oz0-admin --from-literal=password='' ``` Store `master.key` where you keep your other recovery secrets. Without it a restored database cannot be opened. ## Install the chart ```yaml [values.yaml] image: server: {repository: registry.example.com/oz0/orchestrator-zero, tag: "0.9.0"} node: {repository: registry.example.com/oz0/orchestrator-zero-node, tag: "0.9.0"} cluster: {name: prod-eu, historyShards: 1024} database: {existingSecret: oz0-db} masterKey: {existingSecret: oz0-master-key} init: adminPassword: {existingSecret: oz0-admin} ``` ```bash [Terminal] helm install oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml --wait ``` Before the servers start, a hook Job runs `server init --master-key-file ... --exist-ok --temporal-backend `. The first time it creates the cluster with your key, and, only with the embedded backend, Temporal's schemas; with `external` or `cloud` it leaves Temporal's databases alone. Later installs and every upgrade run it again: it only checks that the key unseals the cluster, so a wrong key stops the upgrade before any server restarts. The history shard count can never change after the first install. ## Get operator credentials The first operator's credentials are not kept by the init Job. Issue them in a server pod, which has the database and the master key, and copy them out: ```bash [Terminal] kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- oz0 server operator issue admin \ --master-key-file /etc/orchestrator-zero/master-key/master.key --operator-dir /tmp/operator --advertise 127.0.0.1 kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- tar -C /tmp -cf - operator | tar -xf - kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- rm -rf /tmp/operator export OZ0_CONTEXT=$PWD/operator/context.json ``` The context points at `127.0.0.1`, for port-forwards: ```bash [Terminal] kubectl -n oz0 port-forward svc/oz0-orchestrator-zero-management 8443 & kubectl -n oz0 port-forward svc/oz0-orchestrator-zero-edge 7233 7443 & oz0 cluster info ``` The console is at `https://127.0.0.1:8443` with the user `admin` and the password from the `init.adminPassword` Secret. The chart requires that Secret while `init.enabled` is on: a password the init Job made up would be printed to its log, which log collectors keep, and the Job is deleted once it succeeds. Change the password later with `oz0 user password admin`. ## Add nodes Nodes in the cluster join with a token from a Secret, and each keeps its identity, uv, Python, runtimes and plugins in a volume of its own: ```bash [Terminal] oz0 token create --max-uses 10 --ttl 720h --json # the "token" field kubectl -n oz0 create secret generic oz0-join-token --from-literal=token='K10...' helm upgrade oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml \ --set nodes.enabled=true,nodes.replicas=3,nodes.joinToken.existingSecret=oz0-join-token ``` A token can be used as many times as `--max-uses` says, until its TTL ends: give it enough for the replicas you will add. A replica that lost its volume joins again as a new node. Nodes outside the cluster join through the edge Service: make it a `LoadBalancer` (`service.edge.type`) and add its name with `server.sans`. They need only ports 7233 and 7443 ([Join nodes](https://docs.oz0.ai/fleet/join-nodes)). A node you purge (`oz0 node purge`) empties its volume, but a volume's mount point cannot be removed, so it leaves a marker there, `.oz0-purged`. When the StatefulSet restarts the pod, its entrypoint sees the marker, logs that the node was purged and waits, instead of joining again with the token from the Secret. Scale `nodes.replicas` down to drop the replica. To have it join as a new node instead, delete the marker (`kubectl -n oz0 exec -- rm /var/lib/orchestrator-zero-node/.oz0-purged`), or delete its PersistentVolumeClaim and then its pod. ### Node pods drain before they stop A node pod that is deleted, evicted or rolled drains first, so the work it runs is not lost: 1. Its preStop hook runs `orchestrator-zero-node drain --wait`, in the data directory the image uses (`OZ0_NODE_DATA_DIR`, which `nodes.extraEnv` may set): the node takes no new job and starts no new session, and the hook returns once no session or call is in flight there, or when the node's stop grace is ending. 2. Kubernetes then sends SIGTERM, and the node stops its runtimes and plugins. `nodes.stopGraceSeconds` (600 by default, at most 3600) is the node's stop grace, and the pod's `terminationGracePeriodSeconds` is that and a minute more, which covers the hook and the stop. A session longer than the grace is lost when the grace ends. Before maintenance you plan, drain the node with a deadline first (`oz0 node drain NODE --deadline 2h`, [Patch a node](https://docs.oz0.ai/fleet/manage-nodes#patch-a-node)) and wait for "safe to stop" before you drain the Kubernetes node it runs on. The chart sets no PodDisruptionBudget for nodes. `kubectl drain` evicts every node pod on a Kubernetes node at once, and each drains within its grace while the others take the new work. To keep some nodes taking work while Kubernetes nodes drain one after another, add a PodDisruptionBudget of your own for the node pods, such as `maxUnavailable: 1` that selects the release's node pods by their labels `app.kubernetes.io/instance: ` and `app.kubernetes.io/component: node`; an eviction it blocks waits, and `kubectl drain` tries again. A budget never makes a pod's own drain longer than its grace. ### Sandbox tier 1 in node pods The node image carries bubblewrap, but a pod with Kubernetes' defaults cannot use it: the container runtime's default seccomp profile refuses the namespaces bubblewrap creates, and `/proc` is masked, so bubblewrap cannot turn nested user namespaces off inside its sandbox. Such a node runs harness sessions in sandbox tier 0, and `oz0 node show` says so. Since shell and write sessions run only on tier-1 nodes, a default node pod does not take them. `nodes.sandboxTier1.enabled` gives the node pods what bubblewrap needs, and nothing more:
Setting Why
hostUsers: false A user namespace of the pod's own: root in the pod is no user on the host, and Kubernetes allows an unmasked /proc only then.
procMount: Unmasked bubblewrap writes /proc/sys/user/max_user_namespaces in its sandbox, so a session cannot create user namespaces of its own.
seccompProfile Localhost with the chart's files/seccomp-node-tier1.json (the default), or Unconfined . The runtime's default profile leaves the node at tier 0, so the chart refuses it.
AppArmor Left at the runtime's default unless nodes.sandboxTier1.appArmorProfile sets Localhost (the chart's files/apparmor-node-tier1 ), Unconfined or RuntimeDefault . On hosts that run AppArmor, the default is expected to stop bubblewrap (see the AppArmor settings below).
The pod stays unprivileged: no capability is added, privilege escalation stays off and the root file system read-only. bubblewrap sets no-new-privileges and its own seccomp filter on every session, and Debian's bubblewrap is not setuid. Before you turn it on, complete these checks. #### Check user namespace support Kubernetes user namespaces are required. They are on by default from Kubernetes 1.33 and need containerd 2.0 or CRI-O 1.25 or later, runc 1.2 or crun 1.9 or later, and Linux 6.3 or later on the hosts, with the pods' volumes on file systems that support idmapped mounts (ext4, xfs, btrfs, tmpfs, overlayfs) ([User namespaces](https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/)). #### Install the seccomp profile on every node host Copy `deploy/helm/orchestrator-zero/files/seccomp-node-tier1.json` to `/var/lib/kubelet/seccomp/oz0/node-tier1.json` (or set `nodes.sandboxTier1.seccompProfile.localhostProfile` to where you put it). A pod whose profile is missing does not start. The profile refuses the kernel interfaces the runtime's default keeps from an unprivileged container (kernel modules, the keyrings, `bpf`, `ptrace` and the other calls that read another process's memory or take its file descriptors, `io_uring`, `kexec`, the new mount API and more), apart from creating namespaces and the mount calls bubblewrap makes. It is a list of refusals, not the runtime's list of allowed calls, so it is looser than the default around the node process itself; the sessions in the sandbox still get bubblewrap's own filter. #### Set the namespace's Pod Security level With the `Localhost` seccomp profile, a tier-1 pod met the namespace's baseline level on Kubernetes 1.37 (on a host without AppArmor), which allows an unmasked `/proc` for pods with their own user namespace; the restricted level refuses it. A `Localhost` AppArmor profile is also within baseline. `Unconfined` seccomp or AppArmor needs the privileged level. #### Configure AppArmor on the hosts Where the hosts run AppArmor (Ubuntu, Debian, Container-Optimized OS, and the managed clusters whose nodes run them), expect the runtime's default AppArmor profile to stop bubblewrap: containerd's and CRI-O's default profiles deny every mount and writes under `/proc/sys` outside `/proc/sys/kernel`, and bubblewrap needs both, to mount the sandbox's `/proc` and to write `/proc/sys/user/max_user_namespaces`. Such a pod reports tier 0. There are two ways around it: - The chart's AppArmor profile (within baseline). `deploy/helm/orchestrator-zero/files/apparmor-node-tier1` is the runtime's default with those two denials lifted, plus the user-namespace, umount and pivot_root rules bubblewrap needs. Load it on every host that may run a node pod (`sudo install -D -m 0644 apparmor-node-tier1 /etc/apparmor.d/oz0-node-tier1`, then `sudo apparmor_parser -r -W /etc/apparmor.d/oz0-node-tier1`; it needs AppArmor 4, as on Ubuntu 24.04 and Debian 13), and set `nodes.sandboxTier1.appArmorProfile.type=Localhost` and `localhostProfile=oz0-node-tier1`. A pod whose profile is not loaded on its host does not start. Its `userns` rule also lets bubblewrap create user namespaces where Ubuntu's `kernel.apparmor_restrict_unprivileged_userns=1`. - `appArmorProfile.type=Unconfined`, which needs the privileged Pod Security level. Then, on Ubuntu, `kernel.apparmor_restrict_unprivileged_userns=1` still stops unprivileged user namespaces: set it to `0` on those hosts. Neither has been run on a host with AppArmor yet; the profile has only been parsed by `apparmor_parser`. The AppArmor setting is written as the pod annotation, which Kubernetes 1.29 reads and later versions copy into the `appArmorProfile` field (with a deprecation warning). ```bash [Terminal] helm upgrade oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml --set nodes.sandboxTier1.enabled=true kubectl -n oz0 exec oz0-orchestrator-zero-node-0 -- orchestrator-zero-node sandbox check # Sandbox: tier 1 (bubblewrap 0.12.0) oz0 node show NODE # Sandbox tier 1 (bubblewrap 0.12.0) ``` A pod has no cgroup delegated to it, so sessions in its sandbox run without their memory, process and CPU limits, and `sandbox check` says so on its `Limits:` line; the pod's own `resources` still bound the whole node. Where the pod cannot run bubblewrap, the node reports tier 0 with the reason, and tier-1 work goes to other nodes: run nodes outside the cluster ([Join nodes](https://docs.oz0.ai/fleet/join-nodes)) for it. `nodes.runtimeClassName` runs the node pods under a RuntimeClass of yours, such as one for gVisor or Kata Containers. The chart promises no tier from it: the node reports what its bubblewrap probe finds inside, and node pods under gVisor have not been tested. gVisor does not stand in for tier 1 either: the gVisor sandbox for harness sessions is built on bubblewrap's tier 1, and gVisor for tools needs a container engine, which a node pod does not have ([Sandbox tier 2](https://docs.oz0.ai/fleet/sandbox-tier-2)). ## Previous section links What the chart runs : [open guide](https://docs.oz0.ai/fleet/kubernetes/values#what-the-chart-runs). Values : [open guide](https://docs.oz0.ai/fleet/kubernetes/values#values). Backups : [open guide](https://docs.oz0.ai/fleet/kubernetes/maintenance#backups). Upgrades : [open guide](https://docs.oz0.ai/fleet/kubernetes/maintenance#upgrades). Limits : [open guide](https://docs.oz0.ai/fleet/kubernetes/maintenance#limits). ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.