--- title: "Test and rotate secrets" description: "Test a replacement key before storing it and check the 24-hour rollback window before restoring a value." canonical_url: "https://docs.oz0.ai/admin/secrets/rotation" --- # Test and rotate secrets > Test a replacement key before storing it and check the 24-hour rollback window before restoring a value. For: cluster and tenant administrators. Test a replacement key before storing it and check the 24-hour rollback window before restoring a value. ## Rotate, test and roll back `oz0 secret list`, and the Secrets tab under Credentials in the console, show each secret's version (one more for every set and every rollback), how old its value is and who set it, its expiry, and until when its previous value can be restored: ```text [Terminal] SECRET VERSION AGE SET BY EXPIRES ROLLBACK UNTIL STORE ANTHROPIC_API_KEY 4 2h user:kim 2027-01-07 (every 90d) 2026-10-10 14:02 - GITHUB_TOKEN 1 41d connect:user:ana - - - DB_PASSWORD 3 5d store:bao - - store:bao/oz0/acme/db#password (store version 7) Reminders before an expiry (days): 14, 3, 1 ``` A new value, and a rollback, end a cluster admin's word that the provider account behind a record [serves this tenant alone](https://docs.oz0.ai/spend/models-and-keys/data-rules#what-a-call-may-make-the-provider-read-or-fetch) on each record that reads its key from this secret, since the word was said of the old key. The write itself never waits for that: the gateway stops reading the word within seconds, and management takes it off the record afterwards. ### A provider's key is tested before it is stored `secret set` first sends the new value over TLS to an edge (the role that holds and opens secrets anyway). When a [provider record](https://docs.oz0.ai/spend/models-and-keys#providers-are-records) reads the secret (as its key, or as a SigV4 record's access key ID or session token), the edge makes one call per record that spends nothing: the provider's list of models (`GET /v1/models`) for a key sent in a header, and a token request for a [client secret or a service account key](https://docs.oz0.ai/spend/models-and-keys/authentication#how-calls-are-signed). A value the provider refuses is not stored, and the refusal says which provider answered what, never the value: ```bash [Terminal] printf %s "$NEW_KEY" | oz0 secret set ANTHROPIC_API_KEY # Test with anthropic (key): refused: https://api.anthropic.com refuses the key (401) # Error: the provider refused the value, so it was not stored; ... ``` - A secret no provider record reads (a plugin's token, a database password) calls no provider: the edge answers at once, the value is stored untested, and it does not count against the limit below. - Only a refusal stops it. A provider that does not answer, a server that lists no models (a 404) or a spent account is no answer about the key: the value is stored with a note. `--test` refuses those too, and a value no provider record reads. - SigV4 has no free check, so a SigV4 record's parts are skipped and the record's next call tells. - `--no-test --reason "..."` stores a value untested, and the audit trail records why. - The audit trail says what happened: `secret.test` for each test that reached a provider (its result and each provider's answer, no value), and `secret.set` with `test` set to `passed`, `inconclusive`, `untested`, `skipped` (with the reason) or `none`.
A test lasts ten minutes: the edge gives the CLI a ticket that management checks against the value, so `passed` is what the edge found, never what a client claims. The edge and management may disagree on the time by up to five minutes; a ticket refused as too far ahead or behind means their clocks need NTP. A client that sends no ticket, such as an older CLI or a script calling the admin API, is recorded as `none`. - Who may: the tenant's admins, signed in or with an operator certificate; never an app's API key. At most 20 tests that reach a provider per tenant a minute; past that, `secret set` stores the value untested with a note, and `--test` refuses it until the minute is over. The test is the one call the next job would make with the key, to the provider the key is for ([outbound connections](https://docs.oz0.ai/concepts/outbound-connections)). In the console the Secrets tab tests a new value the same way ("Test it with its providers first", on by default) and shows each provider's answer. ### Roll back within 24 hours The value a set replaces is kept, sealed as it was, for 24 hours. Within them, `secret rollback` swaps it back: ```bash [Terminal] oz0 secret rollback ANTHROPIC_API_KEY --reason "the new key has no quota" # Secret ANTHROPIC_API_KEY of tenant default rolled back: version 3's value is live again, as version 5. # Version 4's value is kept until 2026-10-10 14:02. ``` - At once, everywhere. The gateway reads the key on every call, so the next model call on any edge uses the restored value; the nodes that run a plugin declaring the secret get it within seconds and restart their runtimes, as for any set. An edge that cannot reach management answers from its cache, as for a set, until management is back. - A second rollback undoes the first, and never extends the 24 hours: the replaced value is kept until the time the restored one was. - One previous value only. A second set within the 24 hours destroys the first previous value. `secret set --discard-previous` keeps none (for a key that leaked: rotate it, and revoke the old one at its provider), and `secret delete` destroys the previous value with the secret. - Then destroyed. At the end of the 24 hours an edge blanks the previous value within ten minutes, and a rollback says it is gone from the first second after. Each destruction is audited (`secret.history.destroyed`, with the version and why: `window`, `superseded`, `discarded` or `deleted`), and each rollback (`secret.rollback`). - Never in a backup, never in an edge's cache. Previous values are left out of [backups](https://docs.oz0.ai/fleet/backups#back-up), so a restored cluster cannot roll back to the time before the backup, and out of the disk cache an edge keeps while management is away. ### Expiry and reminders A secret may have an expiry: a date, such as the day a provider's key stops working, or a rotation period that every set moves on. ```bash [Terminal] printf %s "$KEY" | oz0 secret set PARTNER_API_KEY --expires 2027-03-31 printf %s "$KEY" | oz0 secret set ANTHROPIC_API_KEY --expires 90d # every 90 days from each set oz0 secret expiry PARTNER_API_KEY 2027-06-30 # or a period, or none oz0 secret reminders 30 7 1 # the tenant's reminder days; --none for none ``` - Reminders go out 14, 3 and 1 days before the expiry by default (up to five days from 1 to 30 per tenant): the [notification](https://docs.oz0.ai/admin/notifications) event `secret.expiring`, by email to the tenant's admins and to whoever set the secret, and to every target of the tenant's rules for the event (email, webhook, channel). Each goes once per secret, expiry and day. It names the secret, the expiry and the days left; never a value, nor who set it. - Overdue. A secret past its expiry is a finding of `oz0 doctor` (`secret_overdue`) for cluster admins and for the tenant's admins, with the fix, a badge on its row of Credentials' Secrets tab (*overdue*, or *expires in …* from the tenant's first reminder day on), and a card on the Overview. - A reminder, not a lock. An expired secret keeps working until its provider stops taking it. A new value set without `--expires` clears a date (the new key is not the one that expires that day) and moves a period on. Changing the expiry is audited (`secret.expiry`) and changes neither the value nor its version. None of this is a [governance setting](https://docs.oz0.ai/govern/reviewed-changes#four-eyes-for-governance-changes): a rollback, an expiry and the reminder days take no reason and no second admin. ## Sitemap See the full [sitemap](https://docs.oz0.ai/sitemap.md) for all pages.