Deploy with Kubernetes

For: cluster operators.

The chart in deploy/helm/orchestrator-zero runs a cluster on Kubernetes: two or more servers on a PostgreSQL you provide, each running management, the edge and Temporal's services. Nodes can run in the cluster too, or join from outside it.

Choose a task

What you need

• Kubernetes 1.29 or later and Helm (the chart is tested with Helm 4.3).
• PostgreSQL 13 to 16, reachable from the pods, with max_connections of at least 100 per server. The user in the URL should be allowed to create databases; otherwise create oz0, oz0_temporal and oz0_temporal_visibility yourself, named after the database in the URL.
• The images, pushed to a registry your cluster pulls from. No official images are published yet; build them with make images (below).
• A network plugin that enforces NetworkPolicy, if you want the chart's policies to mean anything. Most do; kind's does.

Build the images

make images IMAGE_TAG=0.9.0          # orchestrator-zero:0.9.0 and orchestrator-zero-node:0.9.0, built locally
docker tag orchestrator-zero:0.9.0 registry.example.com/oz0/orchestrator-zero:0.9.0
docker push registry.example.com/oz0/orchestrator-zero:0.9.0   # and the same for orchestrator-zero-node

make images never pushes. Both images are based on debian:trixie-slim pinned by digest and run as user 65532 on a read-only root file system.

The node image works outside Kubernetes too:

docker run -d -e OZ0_SERVER=https://edge.example.com:7443 -e OZ0_JOIN_TOKEN='K10...' \
  -v oz0-node:/var/lib/orchestrator-zero-node orchestrator-zero-node:0.9.0

Its entrypoint copies uv, Python and the runtime into the volume unless they are there already, joins once when the volume holds no identity (with OZ0_JOIN_TOKEN, or OZ0_CA_HASH to wait for an operator's node accept), and runs the node. Keep the volume at /var/lib/orchestrator-zero-node: the runtime's virtual environment refers to its Python by that path.

kubectl create namespace oz0
head -c 32 /dev/urandom | base64 > master.key
kubectl -n oz0 create secret generic oz0-master-key --from-file=master.key
kubectl -n oz0 create secret generic oz0-db \
  --from-literal=url='postgres://oz0:<password>@db.internal:5432/oz0?sslmode=require'
kubectl -n oz0 create secret generic oz0-admin --from-literal=password='<at least 12 characters>'

image:
  server: {repository: registry.example.com/oz0/orchestrator-zero, tag: "0.9.0"}
  node: {repository: registry.example.com/oz0/orchestrator-zero-node, tag: "0.9.0"}
cluster: {name: prod-eu, historyShards: 1024}
database: {existingSecret: oz0-db}
masterKey: {existingSecret: oz0-master-key}
init:
  adminPassword: {existingSecret: oz0-admin}

helm install oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml --wait

kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- oz0 server operator issue admin \
  --master-key-file /etc/orchestrator-zero/master-key/master.key --operator-dir /tmp/operator --advertise 127.0.0.1
kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- tar -C /tmp -cf - operator | tar -xf -
kubectl -n oz0 exec oz0-orchestrator-zero-server-0 -- rm -rf /tmp/operator
export OZ0_CONTEXT=$PWD/operator/context.json

kubectl -n oz0 port-forward svc/oz0-orchestrator-zero-management 8443 &
kubectl -n oz0 port-forward svc/oz0-orchestrator-zero-edge 7233 7443 &
oz0 cluster info

oz0 token create --max-uses 10 --ttl 720h --json   # the "token" field
kubectl -n oz0 create secret generic oz0-join-token --from-literal=token='K10...'
helm upgrade oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml \
  --set nodes.enabled=true,nodes.replicas=3,nodes.joinToken.existingSecret=oz0-join-token

A node you purge (oz0 node purge) empties its volume, but a volume's mount point cannot be removed, so it leaves a marker there, .oz0-purged. When the StatefulSet restarts the pod, its entrypoint sees the marker, logs that the node was purged and waits, instead of joining again with the token from the Secret. Scale nodes.replicas down to drop the replica. To have it join as a new node instead, delete the marker (kubectl -n oz0 exec <pod> -- rm /var/lib/orchestrator-zero-node/.oz0-purged), or delete its PersistentVolumeClaim and then its pod.

Node pods drain before they stop

A node pod that is deleted, evicted or rolled drains first, so the work it runs is not lost:

1. Its preStop hook runs orchestrator-zero-node drain --wait, in the data directory the image uses (OZ0_NODE_DATA_DIR, which nodes.extraEnv may set): the node takes no new job and starts no new session, and the hook returns once no session or call is in flight there, or when the node's stop grace is ending.
2. Kubernetes then sends SIGTERM, and the node stops its runtimes and plugins.

nodes.stopGraceSeconds (600 by default, at most 3600) is the node's stop grace, and the pod's terminationGracePeriodSeconds is that and a minute more, which covers the hook and the stop. A session longer than the grace is lost when the grace ends. Before maintenance you plan, drain the node with a deadline first (oz0 node drain NODE --deadline 2h, Patch a node) and wait for "safe to stop" before you drain the Kubernetes node it runs on.

The chart sets no PodDisruptionBudget for nodes. kubectl drain evicts every node pod on a Kubernetes node at once, and each drains within its grace while the others take the new work. To keep some nodes taking work while Kubernetes nodes drain one after another, add a PodDisruptionBudget of your own for the node pods, such as maxUnavailable: 1 that selects the release's node pods by their labels app.kubernetes.io/instance: <release> and app.kubernetes.io/component: node; an eviction it blocks waits, and kubectl drain tries again. A budget never makes a pod's own drain longer than its grace.

Sandbox tier 1 in node pods

The node image carries bubblewrap, but a pod with Kubernetes' defaults cannot use it: the container runtime's default seccomp profile refuses the namespaces bubblewrap creates, and /proc is masked, so bubblewrap cannot turn nested user namespaces off inside its sandbox. Such a node runs harness sessions in sandbox tier 0, and oz0 node show says so. Since shell and write sessions run only on tier-1 nodes, a default node pod does not take them.

nodes.sandboxTier1.enabled gives the node pods what bubblewrap needs, and nothing more:

The pod stays unprivileged: no capability is added, privilege escalation stays off and the root file system read-only. bubblewrap sets no-new-privileges and its own seccomp filter on every session, and Debian's bubblewrap is not setuid.

Before you turn it on, complete these checks.

Check user namespace support

Kubernetes user namespaces are required. They are on by default from Kubernetes 1.33 and need containerd 2.0 or CRI-O 1.25 or later, runc 1.2 or crun 1.9 or later, and Linux 6.3 or later on the hosts, with the pods' volumes on file systems that support idmapped mounts (ext4, xfs, btrfs, tmpfs, overlayfs) (User namespaces).

Install the seccomp profile on every node host

Copy deploy/helm/orchestrator-zero/files/seccomp-node-tier1.json to /var/lib/kubelet/seccomp/oz0/node-tier1.json (or set nodes.sandboxTier1.seccompProfile.localhostProfile to where you put it). A pod whose profile is missing does not start.

The profile refuses the kernel interfaces the runtime's default keeps from an unprivileged container (kernel modules, the keyrings, bpf, ptrace and the other calls that read another process's memory or take its file descriptors, io_uring, kexec, the new mount API and more), apart from creating namespaces and the mount calls bubblewrap makes. It is a list of refusals, not the runtime's list of allowed calls, so it is looser than the default around the node process itself; the sessions in the sandbox still get bubblewrap's own filter.

Set the namespace's Pod Security level

With the Localhost seccomp profile, a tier-1 pod met the namespace's baseline level on Kubernetes 1.37 (on a host without AppArmor), which allows an unmasked /proc for pods with their own user namespace; the restricted level refuses it. A Localhost AppArmor profile is also within baseline. Unconfined seccomp or AppArmor needs the privileged level.

Configure AppArmor on the hosts

Where the hosts run AppArmor (Ubuntu, Debian, Container-Optimized OS, and the managed clusters whose nodes run them), expect the runtime's default AppArmor profile to stop bubblewrap: containerd's and CRI-O's default profiles deny every mount and writes under /proc/sys outside /proc/sys/kernel, and bubblewrap needs both, to mount the sandbox's /proc and to write /proc/sys/user/max_user_namespaces. Such a pod reports tier 0. There are two ways around it:

• The chart's AppArmor profile (within baseline). deploy/helm/orchestrator-zero/files/apparmor-node-tier1 is the runtime's default with those two denials lifted, plus the user-namespace, umount and pivot_root rules bubblewrap needs. Load it on every host that may run a node pod (sudo install -D -m 0644 apparmor-node-tier1 /etc/apparmor.d/oz0-node-tier1, then sudo apparmor_parser -r -W /etc/apparmor.d/oz0-node-tier1; it needs AppArmor 4, as on Ubuntu 24.04 and Debian 13), and set nodes.sandboxTier1.appArmorProfile.type=Localhost and localhostProfile=oz0-node-tier1. A pod whose profile is not loaded on its host does not start. Its userns rule also lets bubblewrap create user namespaces where Ubuntu's kernel.apparmor_restrict_unprivileged_userns=1.
• appArmorProfile.type=Unconfined, which needs the privileged Pod Security level. Then, on Ubuntu, kernel.apparmor_restrict_unprivileged_userns=1 still stops unprivileged user namespaces: set it to 0 on those hosts.

Neither has been run on a host with AppArmor yet; the profile has only been parsed by apparmor_parser. The AppArmor setting is written as the pod annotation, which Kubernetes 1.29 reads and later versions copy into the appArmorProfile field (with a deprecation warning).

helm upgrade oz0 ./deploy/helm/orchestrator-zero -n oz0 -f values.yaml --set nodes.sandboxTier1.enabled=true
kubectl -n oz0 exec oz0-orchestrator-zero-node-0 -- orchestrator-zero-node sandbox check   # Sandbox: tier 1 (bubblewrap 0.12.0)
oz0 node show NODE                                                                          # Sandbox  tier 1 (bubblewrap 0.12.0)

A pod has no cgroup delegated to it, so sessions in its sandbox run without their memory, process and CPU limits, and sandbox check says so on its Limits: line; the pod's own resources still bound the whole node. Where the pod cannot run bubblewrap, the node reports tier 0 with the reason, and tier-1 work goes to other nodes: run nodes outside the cluster (Join nodes) for it.

nodes.runtimeClassName runs the node pods under a RuntimeClass of yours, such as one for gVisor or Kata Containers. The chart promises no tier from it: the node reports what its bubblewrap probe finds inside, and node pods under gVisor have not been tested. gVisor does not stand in for tier 1 either: the gVisor sandbox for harness sessions is built on bubblewrap's tier 1, and gVisor for tools needs a container engine, which a node pod does not have (Sandbox tier 2).

Previous section links

What the chart runs: open guide.

Values: open guide.

Backups: open guide.

Upgrades: open guide.

Limits: open guide.