Maintain a Kubernetes deployment

For: cluster operators.

Back up persistent state, upgrade the chart and review current limits.

Backups

With backup.enabled, a CronJob runs server backup into a volume and keeps the newest backup.keep archives. The archive holds no master key, and Temporal's own databases are left to PostgreSQL's tools. Copy the archives out of the cluster as well; a volume is lost with it. To take one now:

kubectl -n oz0 create job --from=cronjob/oz0-orchestrator-zero-backup backup-now

Upgrades

Change the image tag and run helm upgrade. A release installed before the chart kept the erasure ledger on a volume needs its server StatefulSet replaced once, since Kubernetes does not let a StatefulSet gain a volume claim template (upgrades). The init hook checks the master key first; then the servers restart one at a time, and the first new one migrates the database under an advisory lock while the others wait (Upgrades).

Nodes in the cluster change with their image tag too: a node in a pod cannot replace its own binary, so node upgrade --binary is for nodes outside the cluster. Their runtimes still roll out from the edge as everywhere else.

A release whose servers had no volume of their own (a chart from before server.persistence) cannot gain one by helm upgrade alone, since Kubernetes never changes a StatefulSet's volume claims: delete the StatefulSet without its pods first (kubectl -n oz0 delete statefulset <release>-server --cascade=orphan), then upgrade, and the servers are replaced one at a time with their volumes.

Limits

• The images are built locally; nothing official is published yet.
• The apt packages in the images (git, ca-certificates, and bubblewrap in the node image) are what Debian's archive held for the pinned base when the image was built.
• Nodes in pods run harness sessions in sandbox tier 0 unless nodes.sandboxTier1 is on and the cluster meets its requirements: user namespaces, the seccomp profile on the hosts, a Pod Security level that allows them, and, on hosts that run AppArmor, the chart's AppArmor profile loaded (or AppArmor unconfined). Tier 1 in a pod runs without its cgroup limits.
• So a node pod takes no shell or write harness session, since those run only on tier-1 nodes, unless the pod runs the chart's tier-1 configuration or a tenant admin opts that node out with a reason (The sandbox opt-out); such a job fails after 10 minutes, naming both, rather than run without a sandbox.
• make e2e-helm tests the default layout (all roles in each server, the embedded Temporal) on kind, with a node pod at tier 0 and then at tier 1 (the Localhost seccomp profile, a namespace that enforces the baseline level, a host without AppArmor).
  The Unconfined variant, the AppArmor profile, AppArmor hosts, managed clusters and RuntimeClasses are checked by make helm-lint only. kind's container runtime applies no AppArmor profile even on a host that runs AppArmor, so a kind run cannot show what AppArmor does to the pod. The separate roles and the other backends are checked by make helm-lint only.