Secrets management: storage, rotation and leaked-secret response

A database password, API token, signing key, and encryption key do different jobs. A stolen database password authorizes access while the server accepts it. A signing key may enable new forgeries. An encryption key may be needed to read years of stored data. Record purpose, owner, issuer, consumers, permissions, storage locations, refresh behavior, and the consequence of compromise before choosing a rotation mechanism.

For a billing API token, draw the path: provider issues token; deployment writes a versioned secret; workers fetch it; each worker caches it; outbound requests carry it; provider verifies it. Copies may also appear in process environments, crash reports, build logs, support exports, and backups. A secret manager governs one part of this path; it does not automatically revoke the provider credential or erase those copies.

Prefer a workload identity and short-lived issued credentials where the target service supports them. That reduces the value and lifetime of copied credentials, but creates dependencies on the issuer, clock, and refresh process. A static integration token may still be necessary; give it the smallest practical scope and an explicit owner rather than treating its existence as a design failure.

The workload needs permission to retrieve only its own secret versions. Granting every application access to the whole store turns one runtime compromise into a fleet-wide compromise. Likewise, a deployment identity that can update secrets should not automatically have permission to export unrelated production values.

Pass credentials through a channel appropriate to the runtime and threat model. Environment variables are convenient but may be inherited by subprocesses or included in diagnostics; files need access controls and cleanup; in-memory values can still appear in dumps. Do not promise absolute secrecy from an attacker who fully controls the process that must use the credential.

Logs should contain secret identifiers, version numbers, and outcomes, not values. Redaction must apply before storage and export, including error paths. Pattern-based scanning finds some formats and historical mistakes, but an unrecognized provider token or transformed credential can evade it. A clean scan is useful evidence about the scanner’s coverage, not a certificate that no secret exists.

How the value reaches the process matters as much as where it is stored. On Kubernetes and other runtimes, prefer mounting from a secret manager or issuing short-lived credentials to copying values into images, Git or long-lived environment variables. The runtime guide at /domains/identity-crypto/secrets/secrets-in-kubernetes-and-runtime/ covers Vault and its Agent Injector, Kubernetes Secrets with KMS v2 encryption, the Secrets Store CSI Driver, External Secrets Operator, SOPS and cloud secret-manager rotation.

Generate every secret, token and key with the operating system’s cryptographically secure random number generator, never with a general-purpose random function, a timestamp or a UUID. Store what you issue as a hash where you only need to check it later, as with API keys: a fast hash such as SHA-256 is enough for a 256-bit random value, unlike passwords, which people choose and which need a slow password hash.

This Python example issues API keys with a recognizable prefix, so users and secret scanners can spot a leaked one, a public key ID for lookup and revocation, and a 256-bit secret that is shown once and stored only as a hash. Verification compares hashes in constant time. It was tested on Python 3.14, including revoked, unknown and altered keys, and secrets that contain underscores.

import hashlib
import hmac
import secrets

PREFIX = "acme_live_"  # a fixed prefix lets users and secret scanners recognize a leaked key


def new_api_key() -> tuple[str, str, str]:
    """Return (key shown once to the user, key ID to store, SHA-256 of the secret to store)."""
    key_id = secrets.token_hex(8)  # public: for lookup, logs and revocation
    secret = secrets.token_urlsafe(32)  # 256 bits from the operating system CSPRNG
    return f"{PREFIX}{key_id}_{secret}", key_id, hashlib.sha256(secret.encode()).hexdigest()


def verify_api_key(presented: str, stored_hash_for) -> str | None:
    """Return the key ID if the presented key is valid. A fast hash is enough for a 256-bit random
    secret; passwords, which people choose, need a slow password hash instead."""
    if not presented.startswith(PREFIX):
        return None
    key_id, _, secret = presented[len(PREFIX):].partition("_")
    stored = stored_hash_for(key_id)  # None if unknown or revoked
    candidate = hashlib.sha256(secret.encode()).hexdigest()
    if stored is None or not hmac.compare_digest(candidate, stored):  # constant-time comparison
        return None
    return key_id

For a provider that supports overlapping tokens, issue B while A remains valid, store B as a new version, direct new deployments and refreshes to B, and observe successful use by every consumer. Then revoke A at the provider and test it. The overlap absorbs deployment and cache delays, but is also a period when both stolen values could work.

For a provider that supports only one password, changing the issuer first can break every cached consumer. Coordinate a maintenance window, support an atomic credential change where available, or redesign the connection mechanism. Do not invent a two-key overlap in documentation when the provider cannot implement it.

Emergency revocation changes the priority: a confirmed compromised credential may need immediate invalidation despite temporary failures. Routine rotation optimizes continuity; incident response limits ongoing access. In either case, define how retries behave so a failure does not print the secret or trigger an unbounded fallback to the old version.

Encryption-key rotation needs a separate read/migrate/retire plan because old ciphertext may require the old key. Never apply a token-revocation checklist directly to data-encryption keys. Also test a restored backup: an old worker image may restart with a revoked token and must refresh safely rather than silently recreating the old configuration.

Treat a secret that reached a public repository, a shared log, a ticket or a chat as compromised from the moment it was exposed, even if it was deleted a minute later. The runbook at /domains/identity-crypto/secrets/leaked-secret-response/ gives the provider-specific steps; work in this order:

• Revoke or rotate the credential at its issuer first, using the emergency path from the rotation plan.
• Read the issuer’s audit or usage logs from the first exposure time onward, and contain any use you cannot explain.
• Find every copy: forks and clones, CI logs, build artifacts, container images, caches, tickets and chat. Deleting the file or rewriting history leaves those copies, so history rewriting is cleanup after revocation, not the remedy.
• Issue a narrowly scoped replacement through the normal rotation path, recording version identifiers rather than values.
• Remove the cause: add the commit-time checks below, and move the consumer to workload identity or short-lived credentials where the target supports them.

GitHub push protection for users is on by default and blocks pushes of supported secret types to public repositories. Push protection for a repository’s own pushes is off by default and needs GitHub Secret Protection; turn it on and restrict who may bypass a block. Add a local pre-commit hook and a CI job that scans the full history: gitleaks git . exits with status 1 when a secret exists anywhere in the history, even after the file was deleted (tested with gitleaks 8.30.1), and trufflehog git file://. --results=verified reports the credentials it could confirm are live.

Scanners recognize known formats and high-entropy strings. An unrecognized provider token or a transformed credential can evade them, so a clean scan is evidence about the scanner’s coverage, not a certificate that no secret exists.

A rotation is complete when intended consumers use the replacement and the retired credential is rejected by the authority that checks it. Updating a secret-manager alias proves neither fact. Use synthetic requests with bounded permissions and record version identifiers only.

Ordinary requests using B must succeed; requests using A after revocation must fail; an instance whose cache exceeds the rollout window must recover or fail closed according to the design. Check the same boundary for scheduled jobs, rarely used recovery paths, and a restored deployment. Set alerts for attempted use of revoked version IDs without collecting credential values.

Repair the code

Insecure randomness lab: Generate unguessable API keys

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Secrets Management Cheat Sheet
2. AWS Secrets Manager: rotation lifecycle
3. GitHub: secret scanning coverage
4. GitHub: about push protection
5. Gitleaks: usage and pre-commit hook
6. OWASP ASVS 5.0: V13 Configuration, secret management. Version 5.0.0 (May 2025).

Reference information

Tags

Secrets managementIncident response

SDLC phase

DevelopmentMaintenance

Role

DevelopersPlatform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-11.5.1 · v5.0.0-13.1.4 · v5.0.0-13.3.1 · v5.0.0-13.3.2 · v5.0.0-13.3.4

Download this guide as Markdown

Continue exploring