Set retention and move tenants

For: cluster operators.

Set history retention and plan tenant moves around the jobs already running.

How long histories are kept

Temporal keeps a closed job's history for its namespace's retention, and then deletes it. Each tenant's namespace keeps histories as long as the tenant's retention says, within what the backend allows:

Every backend keeps at least 1 day; Temporal Cloud at most 90. oz0 retention show, the console's Govern, Privacy › Retention and oz0 doctor show what each namespace keeps, its backend's limits, and what to do when it differs:

Retention:           120 days
Temporal histories:  120 days on acme (embedded, backend default; at least 1 day)

• Embedded and external Temporal. One edge at a time (it holds the lease temporal-retention) sets each namespace's retention when it starts, within seconds of a change to a tenant's retention or legal hold, and every 5 minutes. Each change is in the tenant's audit trail as temporal.retention.applied. An external cluster whose API key may not update namespaces gets a finding with the command to run instead: temporal operator namespace update --namespace acme --retention 120d.
• Temporal Cloud. The server never changes anything on Temporal Cloud. The edge compares each namespace's retention with what the tenant's retention wants and, when they differ, says what to run (in oz0 retention show and as a warning in oz0 doctor): tcld namespace retention set --namespace acme.a1b2c --retention-days 90.
• A namespace two tenants share keeps the longest of their retentions, and says so. A tenant without a rule adds only its minimum, so the other tenant's rule decides. (Two tenants never share a backend namespace today; a clash is refused, as above.)
• Only jobs that close after a change follow it. Temporal fixes when a closed job's history goes at the moment it closes: a longer retention does not keep the histories of jobs that closed before, and a shorter one does not delete them sooner.
• Storage. Temporal keeps every history for as long as the retention says, in its database (your PostgreSQL for the embedded Temporal): a year of retention is a year of histories.
• Legal holds. Temporal deletes per namespace, not per job, so while a tenant is under a legal hold the edge copies each closed job's history into the database before Temporal's retention deletes it: see legal holds and Temporal's histories.

More than one backend, and moving a tenant

A cluster can reach several Temporal backends at once, for example its embedded Temporal and a Temporal Cloud account, and each tenant is on one of them. The backend the --temporal-* flags name is called default; name the others in a YAML file that every edge gets with --temporal-backends-file. Each entry takes the same settings as the flags, with the flag's name less --temporal- (and tls_ for the TLS files). Key files and variables are read only where the edge runs, as with the flags:

# /etc/oz0/temporal-backends.yaml
backends:
  - name: cloud
    kind: cloud                       # external or cloud; the embedded Temporal is only ever the default
    cloud_account: a1b2c
    api_key_dir: /etc/oz0/temporal-keys
  - name: eu-cluster
    kind: external
    address: temporal.eu.internal:7233
    tls: true
    tls_ca_file: /etc/oz0/eu-ca.pem
    create_namespaces: true
    encrypt_payloads: true

oz0 server start ... --temporal-backends-file /etc/oz0/temporal-backends.yaml
oz0 tenant backends      # the backends the edges report, the tenants on each, and the moves

Every edge needs the same file; with regions, every edge of a region, and an entry may say region: eu so that an edge of another region refuses to start with it. When any backend encrypts payloads (Temporal Cloud always does), every tenant's payloads are encrypted on every backend, so a tenant can move between them without a gap. Each backend's namespaces are created or checked as on its own; on Temporal Cloud, create the tenant's namespace there before you move it.

Moving a tenant. A new tenant starts on default. To move one, while it runs jobs:

oz0 tenant move acme cloud                           # its namespace there: acme.<account>
oz0 tenant move acme cloud --namespace prod.a1b2c    # or an existing namespace
oz0 tenant move acme                                 # where it is, and how the move goes

• New work goes to the new backend as soon as an edge finds the tenant's namespace ready there, within seconds: new jobs, flows, the routines' schedules (the edge deletes them on the old backend, and makes them again on the new one) and webhook fires. Until the namespace is ready, the move waits, says why in tenant move acme, and new jobs still start on the old backend.
• Running jobs finish where they started. The tenant's nodes run a second set of workers through the edge while the move lasts, which the edge sends to the old backend; children, approvals, signals and retries of an old job stay on the old backend with it. Nodes never learn which backends they are: no address, no name, no credentials.
• Jobs are found on either backend. job get, traces, approvals, exports and the job list look on the new backend first and on the old one for what it does not have; lists and counts hold both. After the move this still holds for the jobs that ended on the old backend, until the tenant moves again.
• One job ID never runs twice. A start whose job ID still runs on the old backend goes there, where Temporal applies its usual rules.
• The move ends when nothing runs on the old backend: the edge counts every few seconds. At zero it first closes the move, so that no edge starts a job on the old backend any more, waits until every edge has applied that, and counts again: a job an edge started there just before keeps its workers until it ends. This takes about 20 seconds more; tenant move acme shows closed at ... meanwhile, and tenant move acme --wait 10m waits for the end.
  tenant move acme --cancel cancels a move before it ends: new jobs start on the old backend again, and the jobs that started on the new one finish there, as a move back.
• A move has a deadline, 7 days unless you give another (tenant move acme cloud --deadline 48h, or --no-deadline). A job that never ends, such as a session that waits for days, would otherwise hold the move open.
  Past the deadline, once the new backend is ready, the edge closes and ends the move with jobs still running on the old backend: they keep running there, with the nodes' workers for it, and stay reachable (lookups, signals, approvals, the job list), while every new job stays on the new backend. tenant move acme then says done ..., at its deadline, with 2 workflows still on default, and the tenant's backend shows moved; jobs still finish on default.

When the last of them ends, the edge records the move as drained (audit tenant.backend.move.drained) and the nodes stop their workers for the old backend.

Change the deadline of a move that runs from now with tenant move acme --deadline 10m (or --no-deadline).

Until the move is drained, the tenant can move again only back to the old backend, and its namespace mapping cannot change.

• If a backend is down mid-move, the move waits: with the old one down, its jobs pause and the move does not end; with the new one down, a start goes to the old one only when the new one answers that it does not have the job, so that one job ID never runs on both; once the move closed, starts fail until the new one is back. Lists leave out the backend that does not answer.
• The console shows each tenant's backend and the moves under Fleet, Cluster, Temporal backends, where cluster admins start and cancel them.

Temporal's own automated migration to Temporal Cloud moves running workflows too, but is pre-release, needs Temporal's help, and goes one way; a move here works between any two backends, both ways, and needs nothing of Temporal.