---
title: "Maintain a Kubernetes deployment"
description: "Back up persistent state, upgrade the chart and review current limits."
canonical_url: "https://docs.oz0.ai/fleet/kubernetes/maintenance"
---
# Maintain a Kubernetes deployment

> Back up persistent state, upgrade the chart and review current limits.

For: cluster operators.

Back up persistent state, upgrade the chart and review current limits.

## Backups

With `backup.enabled`, a CronJob runs `server backup` into a volume and keeps the newest `backup.keep` archives. The archive holds no master key, and Temporal's own databases are left to PostgreSQL's tools. Copy the archives out of the cluster as well; a volume is lost with it. To take one now:

```bash [Terminal]
kubectl -n oz0 create job --from=cronjob/oz0-orchestrator-zero-backup backup-now
```

## Upgrades

Change the image tag and run `helm upgrade`. A release installed before the chart kept the erasure ledger on a volume needs its server StatefulSet replaced once, since Kubernetes does not let a StatefulSet gain a volume claim template ([upgrades](https://docs.oz0.ai/fleet/upgrades/data-migrations#the-erasure-ledger-on-a-volume-of-its-own)). The init hook checks the master key first; then the servers restart one at a time, and the first new one migrates the database under an advisory lock while the others wait ([Upgrades](https://docs.oz0.ai/fleet/upgrades)).

Nodes in the cluster change with their image tag too: a node in a pod cannot replace its own binary, so `node upgrade --binary` is for nodes outside the cluster. Their runtimes still roll out from the edge as everywhere else.

A release whose servers had no volume of their own (a chart from before `server.persistence`) cannot gain one by `helm upgrade` alone, since Kubernetes never changes a StatefulSet's volume claims: delete the StatefulSet without its pods first (`kubectl -n oz0 delete statefulset <release>-server --cascade=orphan`), then upgrade, and the servers are replaced one at a time with their volumes.

## Limits

- The images are built locally; nothing official is published yet.
- The apt packages in the images (`git`, `ca-certificates`, and `bubblewrap` in the node image) are what Debian's archive held for the pinned base when the image was built.
- Nodes in pods run harness sessions in sandbox tier 0 unless `nodes.sandboxTier1` is on and the cluster meets [its requirements](https://docs.oz0.ai/fleet/kubernetes#sandbox-tier-1-in-node-pods): user namespaces, the seccomp profile on the hosts, a Pod Security level that allows them, and, on hosts that run AppArmor, the chart's AppArmor profile loaded (or AppArmor unconfined). Tier 1 in a pod runs without its cgroup limits.
- So a node pod takes no shell or write harness session, since those run only on tier-1 nodes, unless the pod runs the chart's tier-1 configuration or a tenant admin opts that node out with a reason ([The sandbox opt-out](https://docs.oz0.ai/fleet/nodes#the-sandbox-opt-out)); such a job fails after 10 minutes, naming both, rather than run without a sandbox.
- `make e2e-helm` tests the default layout (all roles in each server, the embedded Temporal) on kind, with a node pod at tier 0 and then at tier 1 (the `Localhost` seccomp profile, a namespace that enforces the baseline level, a host without AppArmor).<br />

The `Unconfined` variant, the AppArmor profile, AppArmor hosts, managed clusters and RuntimeClasses are checked by `make helm-lint` only. kind's container runtime applies no AppArmor profile even on a host that runs AppArmor, so a kind run cannot show what AppArmor does to the pod. The separate roles and the other backends are checked by `make helm-lint` only.


## Sitemap

See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.