Skip to main content

​
Secret Rotation Runbook

Every secret configured for apps/platform (see apps/platform/.env.example), plus the compliance signing key, which is deliberately not committed to .env.example since it must never take the placeholder value changethis. For each: what it protects, rotation steps, and blast radius if it leaks or rotation goes wrong. Secrets live in Railway environment variables for staging/production (see docs/project/deployment.md) and in a local .env for development. Rotating a secret always means: generate the new value, set it in Railway, redeploy, confirm health, then invalidate the old value at the provider if the provider supports that (API keys) — never the other way around, or you create a window with no working credential.

​
Database — POSTGRES_PASSWORD (or DATABASE_URL)

Protects: the primary Postgres database — every run, action, ledger row, policy, and user record. Rotation steps:
  1. In the Postgres provider (Railway-managed or external), create a new password for the POSTGRES_USER role without revoking the old one yet.
  2. Update POSTGRES_PASSWORD (or the full DATABASE_URL, which takes precedence per app/core/config.py) in the platform’s Railway environment.
  3. Redeploy the platform service and confirm GET /api/v1/utils/health-check/ returns healthy and a real query path (e.g. list runs) succeeds.
  4. Revoke the old password at the database.
Blast radius: a leaked database credential is a full-tenant-data compromise — every organization’s runs, ledger, and policies are reachable. Treat any suspected leak as SEV1 per docs/project/incident-protocol.md, rotate immediately, and check pg_stat_activity (or the provider’s connection audit log) for connections you don’t recognize before and after rotation.

​
Compliance Signing Key — COMPLIANCE_SIGNING_KEY_ED25519 (+ COMPLIANCE_SIGNING_KEY_ID)

Protects: the Ed25519 private key ComplianceSigner (apps/platform/app/services/compliance_signing.py) uses to sign compliance export manifests. It is what makes an export tamper-evident and auditor-verifiable. Rotation steps:
  1. Generate a new key pair:
  2. Choose a new COMPLIANCE_SIGNING_KEY_ID — do not reuse the old ID, since exports already signed under the old ID must remain verifiable against the old public key.
  3. Publish the new key’s public key (ComplianceSigner.get_public_key_pem()) wherever existing public keys are distributed for verification, keyed by the new COMPLIANCE_SIGNING_KEY_ID. Do not delete the old public key — past exports still need it.
  4. Set COMPLIANCE_SIGNING_KEY_ED25519 and COMPLIANCE_SIGNING_KEY_ID to the new values in Railway and redeploy.
  5. Confirm a fresh compliance export is signed under the new key ID.
Blast radius: if this key leaks, an attacker can forge compliance export signatures — anything an auditor would treat as tamper-evident becomes untrustworthy for exports signed after the leak. It does not expose customer data directly, but it is a trust-chain compromise: rotate immediately and treat every export produced between suspected leak and rotation as unverifiable. Never overwrite an existing COMPLIANCE_SIGNING_KEY_ID with a new key — that breaks verification for every export already issued under that ID.

​
Clerk — CLERK_SECRET_KEY, CLERK_PUBLISHABLE_KEY, CLERK_WEBHOOK_SECRET

Protects: authentication. CLERK_SECRET_KEY backs server-side Clerk API calls (apps/platform/app/core/clerk.py); CLERK_PUBLISHABLE_KEY is used client-side and is not sensitive by design; CLERK_WEBHOOK_SECRET verifies that inbound webhooks (apps/platform/app/api/routes/clerk_webhooks.py) actually came from Clerk. Rotation steps:
  1. In the Clerk dashboard, generate a new secret key (and webhook signing secret, if rotating that too) for the instance.
  2. Set CLERK_SECRET_KEY (and/or CLERK_WEBHOOK_SECRET) in Railway and redeploy.
  3. Confirm sign-in and at least one webhook-driven flow (e.g. a user creation event) still work.
  4. Revoke the old key in the Clerk dashboard.
Rotating CLERK_PUBLISHABLE_KEY also requires updating the console’s build (apps/console-sentinel) and redeploying it, since that key is baked into the client bundle. Blast radius: a leaked CLERK_SECRET_KEY lets an attacker act as the Clerk backend against your instance — impersonate the platform’s own calls to Clerk (user lookups, session verification). A leaked CLERK_WEBHOOK_SECRET lets an attacker forge webhook events (e.g. fake user creation/deletion) since verify_webhook_signature would accept them. Both are auth-adjacent — treat as SEV1/SEV2 per the incident protocol.

​
Resend — RESEND_API_KEY

Protects: outbound transactional email (approval notifications, password resets) sent via apps/platform/app/utils.py. If unset, email sending is skipped with a warning, not a failure. Rotation steps:
  1. Create a new API key in the Resend dashboard.
  2. Set RESEND_API_KEY in Railway and redeploy.
  3. Send a test email (e.g. trigger a password reset) to confirm delivery.
  4. Revoke the old key in Resend.
Blast radius: a leaked key lets an attacker send email as EMAILS_FROM_EMAIL (noreply@agentsentinel.dev) through your Resend account — reputational and phishing risk, and potential quota/billing abuse. It does not expose stored data. SEV2 unless there’s evidence of abuse already in progress.

​
Gemini — GEMINI_API_KEY

Protects: nothing customer-sensitive by itself, but it authorizes calls to Google’s Gemini API, billed to your account. Used for prose policy compilation (app/services/policy_compiler.py), the MCP policy advisor (app/services/mcp_advisor.py), intervention enrichment (app/services/intervention_enricher.py), and eval scenario generation (app/services/gemini_scenario_generator.py). All of these degrade gracefully (log and skip) when the key is absent — rotating it never causes an outage, at worst a temporary loss of these optional features. Rotation steps:
  1. Create a new key at https://aistudio.google.com/apikey.
  2. Set GEMINI_API_KEY in Railway and redeploy.
  3. Delete the old key in AI Studio.
Blast radius: a leaked key is a billing/quota risk (an attacker can run up usage against your account) and, since prompts sent to these features may include intervention/policy context, a potential data-exposure path to Google’s API if the key is used outside your control. Not customer-data critical on its own, but rotate promptly and check usage in AI Studio for anomalies.

​
Platform Session Signing — SECRET_KEY

Protects: HS256 signing for session/access JWTs and password-reset tokens (app/core/security.py, app/api/deps.py, app/utils.py) and, in local/offline-auth mode, dev tokens (app/core/clerk_dev.py). Ships with a random default in code, but .env.example uses the placeholder changethis — Settings._enforce_non_default_secrets refuses to start in staging/production if it’s still changethis. Rotation steps:
  1. Generate a new value, e.g. python -c "import secrets; print(secrets.token_urlsafe(32))".
  2. Set SECRET_KEY in Railway and redeploy.
Blast radius: rotating it immediately invalidates every existing session and password-reset token — all logged-in users are signed out. A leaked SECRET_KEY lets an attacker forge valid session tokens for any user. Rotate immediately on suspected leak (SEV1) and accept the forced sign-out as the cost of containment.

​
First Superuser — FIRST_SUPERUSER_PASSWORD

Protects: the bootstrap admin account (FIRST_SUPERUSER). Also refused at startup if left as changethis outside local. Rotation steps:
  1. Change the password through the platform’s own user-management path (or update FIRST_SUPERUSER_PASSWORD and re-run the seed step if the account is recreated from env on deploy).
  2. If rotating in Railway, redeploy after setting the new value.
Blast radius: this account has superuser access — treat a leak as SEV1, same as any other admin credential compromise.

​
SMTP — SMTP_PASSWORD

Protects: local-development email delivery only in this deployment (Mailcatcher in local, unused in staging/production where RESEND_API_KEY is the real email path — see .env.example’s local-development block). Rotation steps: rotate at the SMTP provider if one is ever configured for staging/production; for local Mailcatcher, there’s nothing to rotate. Blast radius: minimal in the current deployment (local dev only); would follow the same email-abuse blast radius as Resend if a real SMTP provider is ever wired up for a non-local environment.

​
Sentry — SENTRY_DSN

Protects: nothing by itself (a DSN is not a secret Sentry treats as sensitive — it’s meant to be embedded in client code), but rotating it stops error reports from reaching your Sentry project, which matters if a former collaborator’s Sentry access needs to be cut off. Rotation steps: generate a new DSN in the Sentry project settings, set SENTRY_DSN in Railway, redeploy. Blast radius: low. Worst case is noise (someone sends fake error events to your DSN) or a gap in error visibility during rotation.