> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentsentinel.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Rotation runbook

# Secret Rotation Runbook

Every secret configured for `apps/platform` (see
[`apps/platform/.env.example`](../../apps/platform/.env.example)), plus the
compliance signing key, which is deliberately not committed to `.env.example`
since it must never take the placeholder value `changethis`. For each: what
it protects, rotation steps, and blast radius if it leaks or rotation goes
wrong.

Secrets live in Railway environment variables for staging/production (see
[`docs/project/deployment.md`](deployment.md)) and in a local `.env` for
development. Rotating a secret always means: generate the new value, set it
in Railway, redeploy, confirm health, then invalidate the old value at the
provider if the provider supports that (API keys) — never the other way
around, or you create a window with no working credential.

## Database — `POSTGRES_PASSWORD` (or `DATABASE_URL`)

**Protects:** the primary Postgres database — every run, action, ledger row,
policy, and user record.

**Rotation steps:**

1. In the Postgres provider (Railway-managed or external), create a new
   password for the `POSTGRES_USER` role without revoking the old one yet.
2. Update `POSTGRES_PASSWORD` (or the full `DATABASE_URL`, which takes
   precedence per `app/core/config.py`) in the platform's Railway
   environment.
3. Redeploy the platform service and confirm
   `GET /api/v1/utils/health-check/` returns healthy and a real query path
   (e.g. list runs) succeeds.
4. Revoke the old password at the database.

**Blast radius:** a leaked database credential is a full-tenant-data
compromise — every organization's runs, ledger, and policies are reachable.
Treat any suspected leak as SEV1 per
[`docs/project/incident-protocol.md`](incident-protocol.md), rotate
immediately, and check `pg_stat_activity` (or the provider's connection
audit log) for connections you don't recognize before and after rotation.

## Compliance Signing Key — `COMPLIANCE_SIGNING_KEY_ED25519` (+ `COMPLIANCE_SIGNING_KEY_ID`)

**Protects:** the Ed25519 private key `ComplianceSigner`
(`apps/platform/app/services/compliance_signing.py`) uses to sign compliance
export manifests. It is what makes an export tamper-evident and
auditor-verifiable.

**Rotation steps:**

1. Generate a new key pair:
   ```bash theme={null}
   python -c 'from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey; import base64; k = Ed25519PrivateKey.generate(); print(base64.b64encode(k.private_bytes_raw()).decode())'
   ```
2. Choose a new `COMPLIANCE_SIGNING_KEY_ID` — do not reuse the old ID, since
   exports already signed under the old ID must remain verifiable against
   the old public key.
3. Publish the new key's public key (`ComplianceSigner.get_public_key_pem()`)
   wherever existing public keys are distributed for verification, keyed by
   the new `COMPLIANCE_SIGNING_KEY_ID`. Do not delete the old public key —
   past exports still need it.
4. Set `COMPLIANCE_SIGNING_KEY_ED25519` and `COMPLIANCE_SIGNING_KEY_ID` to
   the new values in Railway and redeploy.
5. Confirm a fresh compliance export is signed under the new key ID.

**Blast radius:** if this key leaks, an attacker can forge compliance export
signatures — anything an auditor would treat as tamper-evident becomes
untrustworthy for exports signed after the leak. It does not expose customer
data directly, but it is a trust-chain compromise: rotate immediately and
treat every export produced between suspected leak and rotation as
unverifiable. Never overwrite an existing `COMPLIANCE_SIGNING_KEY_ID` with a
new key — that breaks verification for every export already issued under
that ID.

## Clerk — `CLERK_SECRET_KEY`, `CLERK_PUBLISHABLE_KEY`, `CLERK_WEBHOOK_SECRET`

**Protects:** authentication. `CLERK_SECRET_KEY` backs server-side Clerk API
calls (`apps/platform/app/core/clerk.py`); `CLERK_PUBLISHABLE_KEY` is used
client-side and is not sensitive by design; `CLERK_WEBHOOK_SECRET` verifies
that inbound webhooks (`apps/platform/app/api/routes/clerk_webhooks.py`)
actually came from Clerk.

**Rotation steps:**

1. In the Clerk dashboard, generate a new secret key (and webhook signing
   secret, if rotating that too) for the instance.
2. Set `CLERK_SECRET_KEY` (and/or `CLERK_WEBHOOK_SECRET`) in Railway and
   redeploy.
3. Confirm sign-in and at least one webhook-driven flow (e.g. a user
   creation event) still work.
4. Revoke the old key in the Clerk dashboard.

Rotating `CLERK_PUBLISHABLE_KEY` also requires updating the console's build
(`apps/console-sentinel`) and redeploying it, since that key is baked into
the client bundle.

**Blast radius:** a leaked `CLERK_SECRET_KEY` lets an attacker act as the
Clerk backend against your instance — impersonate the platform's own calls
to Clerk (user lookups, session verification). A leaked
`CLERK_WEBHOOK_SECRET` lets an attacker forge webhook events (e.g. fake user
creation/deletion) since `verify_webhook_signature` would accept them. Both
are auth-adjacent — treat as SEV1/SEV2 per the incident protocol.

## Resend — `RESEND_API_KEY`

**Protects:** outbound transactional email (approval notifications, password
resets) sent via `apps/platform/app/utils.py`. If unset, email sending is
skipped with a warning, not a failure.

**Rotation steps:**

1. Create a new API key in the Resend dashboard.
2. Set `RESEND_API_KEY` in Railway and redeploy.
3. Send a test email (e.g. trigger a password reset) to confirm delivery.
4. Revoke the old key in Resend.

**Blast radius:** a leaked key lets an attacker send email as
`EMAILS_FROM_EMAIL` (`noreply@agentsentinel.dev`) through your Resend
account — reputational and phishing risk, and potential quota/billing abuse.
It does not expose stored data. SEV2 unless there's evidence of abuse
already in progress.

## Gemini — `GEMINI_API_KEY`

**Protects:** nothing customer-sensitive by itself, but it authorizes calls
to Google's Gemini API, billed to your account. Used for prose policy
compilation (`app/services/policy_compiler.py`), the MCP policy advisor
(`app/services/mcp_advisor.py`), intervention enrichment
(`app/services/intervention_enricher.py`), and eval scenario generation
(`app/services/gemini_scenario_generator.py`). All of these degrade
gracefully (log and skip) when the key is absent — rotating it never causes
an outage, at worst a temporary loss of these optional features.

**Rotation steps:**

1. Create a new key at [https://aistudio.google.com/apikey](https://aistudio.google.com/apikey).
2. Set `GEMINI_API_KEY` in Railway and redeploy.
3. Delete the old key in AI Studio.

**Blast radius:** a leaked key is a billing/quota risk (an attacker can run
up usage against your account) and, since prompts sent to these features may
include intervention/policy context, a potential data-exposure path to
Google's API if the key is used outside your control. Not customer-data
critical on its own, but rotate promptly and check usage in AI Studio for
anomalies.

## Platform Session Signing — `SECRET_KEY`

**Protects:** HS256 signing for session/access JWTs and password-reset
tokens (`app/core/security.py`, `app/api/deps.py`, `app/utils.py`) and, in
local/offline-auth mode, dev tokens (`app/core/clerk_dev.py`). Ships with a
random default in code, but `.env.example` uses the placeholder
`changethis` — `Settings._enforce_non_default_secrets` refuses to start in
staging/production if it's still `changethis`.

**Rotation steps:**

1. Generate a new value, e.g. `python -c "import secrets; print(secrets.token_urlsafe(32))"`.
2. Set `SECRET_KEY` in Railway and redeploy.

**Blast radius:** rotating it immediately invalidates every existing session
and password-reset token — all logged-in users are signed out. A leaked
`SECRET_KEY` lets an attacker forge valid session tokens for any user.
Rotate immediately on suspected leak (SEV1) and accept the forced
sign-out as the cost of containment.

## First Superuser — `FIRST_SUPERUSER_PASSWORD`

**Protects:** the bootstrap admin account (`FIRST_SUPERUSER`). Also refused
at startup if left as `changethis` outside local.

**Rotation steps:**

1. Change the password through the platform's own user-management path (or
   update `FIRST_SUPERUSER_PASSWORD` and re-run the seed step if the account
   is recreated from env on deploy).
2. If rotating in Railway, redeploy after setting the new value.

**Blast radius:** this account has superuser access — treat a leak as
SEV1, same as any other admin credential compromise.

## SMTP — `SMTP_PASSWORD`

**Protects:** local-development email delivery only in this deployment
(Mailcatcher in local, unused in staging/production where `RESEND_API_KEY`
is the real email path — see `.env.example`'s local-development block).

**Rotation steps:** rotate at the SMTP provider if one is ever configured
for staging/production; for local Mailcatcher, there's nothing to rotate.

**Blast radius:** minimal in the current deployment (local dev only); would
follow the same email-abuse blast radius as Resend if a real SMTP provider
is ever wired up for a non-local environment.

## Sentry — `SENTRY_DSN`

**Protects:** nothing by itself (a DSN is not a secret Sentry treats as
sensitive — it's meant to be embedded in client code), but rotating it stops
error reports from reaching your Sentry project, which matters if a former
collaborator's Sentry access needs to be cut off.

**Rotation steps:** generate a new DSN in the Sentry project settings, set
`SENTRY_DSN` in Railway, redeploy.

**Blast radius:** low. Worst case is noise (someone sends fake error events
to your DSN) or a gap in error visibility during rotation.