--- title: "Payload storage" description: "Keep large payloads and plugin artifacts in an S3-compatible bucket instead of PostgreSQL." canonical_url: "https://docs.oz0.ai/fleet/payload-storage" --- # Payload storage > Keep large payloads and plugin artifacts in an S3-compatible bucket instead of PostgreSQL. For: cluster operators. Choose where the edges store large payloads and plugin artifacts. Give storage access to the edges that use it. Two kinds of data can grow large: the payloads that Temporal's External Storage moves out of a job's history (every value over 256 KiB, such as a long conversation or a big tool result, up to 32 MiB each), and the artifacts of installed plugins. By default both are rows in PostgreSQL. With `--payload-store` they go to an S3-compatible bucket instead: Amazon S3, or any server that speaks the S3 API, such as SeaweedFS, Ceph, Garage or Cloudflare R2. Only servers that run the edge role reach the bucket. Nodes never do: they hand payloads to the edge and fetch artifacts from it, as before. Management, when it runs apart from the edges, does not reach it either: it hands each artifact to an edge that has the bucket. ## Turn it on Give every server that runs the edge role the same bucket (with [regions](https://docs.oz0.ai/fleet/regions), every edge of a region, and each region a bucket of its own; plugin artifacts are then copied to each region's bucket): ```bash [Terminal] oz0 server start ... \ --payload-store 's3://oz0-payloads/cluster-1?region=eu-north-1&sse=aws:kms' \ --payload-store-credentials-file /etc/oz0/s3-credentials ``` The URL names the bucket, a prefix for every key (`cluster-1/` here), and these settings:
Setting What it does
region The region requests are signed for. Default us-east-1 .
endpoint The S3 API's address, such as https://minio.storage:9000 . Default: AWS's for the region.
path-style true puts the bucket in the path ( https://host/bucket/key ), as most S3-compatible servers need: the default with an endpoint . AWS's own endpoints are addressed by host name ( https://bucket.s3.<region>.amazonaws.com ).
sse Server-side encryption asked for on every write: AES256 or aws:kms . Without it, the bucket's default applies.
kms-key-id The KMS key for sse=aws:kms ; without it, the account's default key.
ca-file A PEM file the endpoint's certificate is checked against, for a server with a private CA.
part-size The part size of multipart uploads, in bytes (default 8 MiB; at least 5 MiB). Objects over twice that go in parts.
The URL never holds credentials. The edge reads them, in this order, from: 1. `--payload-store-credentials-file`: an AWS shared credentials file, its `[default]` section or the `profile=` the URL names. The edge reads it again when it changes, so you can rotate the key without a restart. 2. `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and, for temporary credentials, `AWS_SESSION_TOKEN`. 3. `AWS_SHARED_CREDENTIALS_FILE`, then `~/.aws/credentials`. ```ini [s3-credentials] [default] aws_access_key_id = AKIA... aws_secret_access_key = ... ``` The credentials need `s3:PutObject`, `s3:GetObject`, `s3:DeleteObject`, `s3:ListBucket` and, for multipart uploads, `s3:AbortMultipartUpload` on the bucket and its prefix. At start the edge writes, reads and deletes an object under `oz0-check/` and refuses to start with S3's own answer when one of them fails, so a wrong key, bucket or policy shows up at once. An endpoint over plain HTTP to another host than this one works, with a warning: payloads would cross the network in the clear. With the [roles apart](https://docs.oz0.ai/fleet/separate-roles), an edge with `--payload-store` needs no `OZ0_PAYLOAD_DB`. Management takes no `--payload-store`: when an edge reports a bucket, management keeps new artifacts there through it. In Kubernetes, set the chart's `payloadStore.url` and put the credentials file in a Secret named by `payloadStore.credentials.existingSecret` (see [Kubernetes](https://docs.oz0.ai/fleet/kubernetes)). ## What goes where
Data In the bucket In PostgreSQL
A payload <prefix>payloads/<tenant>/<sha256> nothing
An artifact <prefix>artifacts/sha256/<hex> its row in blobs , without the bytes: the index that installed plugins and the collector use
Each object is named by the SHA-256 of its bytes, and the edge checks what it reads back against the name. A name that is not a SHA-256 in lowercase hex is refused before the bucket is asked, whoever sends it. With [encrypted payloads](https://docs.oz0.ai/fleet/temporal-backends/encryption#encrypted-payloads) the bucket holds ciphertext: External Storage runs after the tenant's key has encrypted the payload, so the bucket's own encryption is a second layer. Payloads and artifacts stored in PostgreSQL before you turned the bucket on stay there and are still read from there; nothing has to move. ## Retention A payload is kept for 90 days after a job last stored it. Storing a payload that is in the bucket already writes it again only when its copy is more than a day old, so its `Last-Modified` is when a job last used it. Every edge deletes, each hour, the payloads under `payloads/` that are older than 90 days, as it deletes old rows. Artifacts are collected by management as before (`plugin gc`), which deletes the object with the row. The same pass deletes the artifacts' objects that no row names, once they are older than the grace period: an install that fails after storing its artifact rolls its row back but leaves the object. `plugin gc --dry-run` lists them with the rest. ## Backups `server backup` holds the artifacts' rows, not their bytes, and no payloads. Back the bucket up with its own tools: versioning, replication to another region, or a copy with the provider's tools. A restore needs the database and the bucket from about the same time. ## Tested with `make test` runs the client against an S3 server in memory that checks every request's signature, and against SeaweedFS 4.48 when `scripts/s3-tools.sh` has fetched it. `make e2e-roles-backends` runs management and two edges apart with a bucket instead of a payload database: a plugin's artifact, and a 300 KiB result that reads back through the edge. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.