Payload storage

For: cluster operators.

Choose where the edges store large payloads and plugin artifacts. Give storage access to the edges that use it.

Two kinds of data can grow large: the payloads that Temporal's External Storage moves out of a job's history (every value over 256 KiB, such as a long conversation or a big tool result, up to 32 MiB each), and the artifacts of installed plugins. By default both are rows in PostgreSQL. With --payload-store they go to an S3-compatible bucket instead: Amazon S3, or any server that speaks the S3 API, such as SeaweedFS, Ceph, Garage or Cloudflare R2.

Only servers that run the edge role reach the bucket. Nodes never do: they hand payloads to the edge and fetch artifacts from it, as before. Management, when it runs apart from the edges, does not reach it either: it hands each artifact to an edge that has the bucket.

Turn it on

Give every server that runs the edge role the same bucket (with regions, every edge of a region, and each region a bucket of its own; plugin artifacts are then copied to each region's bucket):

oz0 server start ... \
  --payload-store 's3://oz0-payloads/cluster-1?region=eu-north-1&sse=aws:kms' \
  --payload-store-credentials-file /etc/oz0/s3-credentials

The URL names the bucket, a prefix for every key (cluster-1/ here), and these settings:

The URL never holds credentials. The edge reads them, in this order, from:

1. --payload-store-credentials-file: an AWS shared credentials file, its [default] section or the profile= the URL names. The edge reads it again when it changes, so you can rotate the key without a restart.
2. AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and, for temporary credentials, AWS_SESSION_TOKEN.
3. AWS_SHARED_CREDENTIALS_FILE, then ~/.aws/credentials.

[default]
aws_access_key_id = AKIA...
aws_secret_access_key = ...

The credentials need s3:PutObject, s3:GetObject, s3:DeleteObject, s3:ListBucket and, for multipart uploads, s3:AbortMultipartUpload on the bucket and its prefix. At start the edge writes, reads and deletes an object under oz0-check/ and refuses to start with S3's own answer when one of them fails, so a wrong key, bucket or policy shows up at once. An endpoint over plain HTTP to another host than this one works, with a warning: payloads would cross the network in the clear.

With the roles apart, an edge with --payload-store needs no OZ0_PAYLOAD_DB. Management takes no --payload-store: when an edge reports a bucket, management keeps new artifacts there through it.

In Kubernetes, set the chart's payloadStore.url and put the credentials file in a Secret named by payloadStore.credentials.existingSecret (see Kubernetes).

What goes where

Each object is named by the SHA-256 of its bytes, and the edge checks what it reads back against the name. A name that is not a SHA-256 in lowercase hex is refused before the bucket is asked, whoever sends it. With encrypted payloads the bucket holds ciphertext: External Storage runs after the tenant's key has encrypted the payload, so the bucket's own encryption is a second layer.

Payloads and artifacts stored in PostgreSQL before you turned the bucket on stay there and are still read from there; nothing has to move.

Retention

A payload is kept for 90 days after a job last stored it. Storing a payload that is in the bucket already writes it again only when its copy is more than a day old, so its Last-Modified is when a job last used it. Every edge deletes, each hour, the payloads under payloads/ that are older than 90 days, as it deletes old rows.

Artifacts are collected by management as before (plugin gc), which deletes the object with the row. The same pass deletes the artifacts' objects that no row names, once they are older than the grace period: an install that fails after storing its artifact rolls its row back but leaves the object. plugin gc --dry-run lists them with the rest.

Backups

server backup holds the artifacts' rows, not their bytes, and no payloads. Back the bucket up with its own tools: versioning, replication to another region, or a copy with the provider's tools. A restore needs the database and the bucket from about the same time.

Tested with

make test runs the client against an S3 server in memory that checks every request's signature, and against SeaweedFS 4.48 when scripts/s3-tools.sh has fetched it. make e2e-roles-backends runs management and two edges apart with a bucket instead of a payload database: a plugin's artifact, and a 300 KiB result that reads back through the edge.