---
key: "secret-handling"
title: "Secrets management: storage, rotation and leaked-secret response"
url: "https://softwaresecurity.com/domains/identity-crypto/secrets/secrets-lifecycle/"
api: "https://softwaresecurity.com/api/v1/articles/secret-handling.json"
domain: "identity-crypto"
topic: "Identity \u0026 cryptography"
type: "Guide"
identifiers: ["CWE-798","CWE-522","CWE-532","A04:2025","A09:2025","V11.5.1","V13.1.4","V13.3.1","V13.3.2","V13.3.4"]
tags: ["Secrets management","Incident response"]
stages: ["build","operate","respond"]
roles: ["developer","platform-sre"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Secrets management: storage, rotation and leaked-secret response

Follow one application credential from issuance through caching, rotation, revocation and restored backups, prove that its old value stops working, and respond correctly when a secret leaks.

Canonical URL: https://softwaresecurity.com/domains/identity-crypto/secrets/secrets-lifecycle/

## In short

Keep each secret in a secret manager that only its consuming workload can read, prefer short-lived credentials from workload identity, and log identifiers, never values. Rotate by issuing the new value, moving every consumer, then revoking the old value at the issuer and proving it fails. If a secret leaks, revoke it at the issuer first, check its use since exposure, then clean up copies; deleting it from Git history is not a fix.

- A secret is a capability: record its owner, issuer, consumers, scope and what a compromise would allow.
- A rotation is complete only when every consumer uses the new value and the issuer rejects the old one.
- Overlap two values for at least the longest consumer cache time plus deployment time, where the provider allows it.
- After a leak: revoke first, investigate use second, clean up history and copies last.
- Block secrets at commit and push time and scan full history in CI; a clean scan covers only formats the scanner knows.
- Prefer workload identity and short-lived tokens to long-lived static secrets.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Generated secrets | At least 128 bits from a CSPRNG | HMAC keys at least as long as the hash output, such as 256 bits for HMAC-SHA256. Never derive a secret from a timestamp, username or other guessable input. | ASVS 5.0 V11.5.1; RFC 2104 |
| Storage and access | Secret manager; each workload reads only its own secrets | Deployment identities that write secrets should not read unrelated ones. | ASVS 5.0 V13.3.1 and V13.3.2 |
| Rotation schedule | Documented per secret, plus immediate rotation on suspected exposure | Base the interval on the secret’s scope and exposure, not a single global number. | ASVS 5.0 V13.1.4 and V13.3.4 |
| Overlap window | Longest consumer cache lifetime plus deployment time | Only where the issuer supports two valid values; otherwise plan a coordinated cutover. | This guide |
| Logging | Secret identifiers, versions and outcomes only | Redact before storage and export, including error paths. | OWASP Secrets Management Cheat Sheet |
| Commit-time protection | GitHub push protection on for repositories, plus a pre-commit or CI scanner | Push protection for users is on by default only for pushes to public repositories. | GitHub documentation: push protection |

## Verify it

1. After a rotation, call the issuer with the old value and confirm it is rejected; call with the new value and confirm success.
2. List who and what can read each production secret, and confirm each workload reads only its own.
3. Search recent logs, crash reports and CI output for the secret’s value pattern and confirm only identifiers appear.
4. Run `gitleaks git .` or an equivalent history scan in CI and confirm the job fails on a planted test secret.
5. Restore an old backup in isolation and confirm it fetches the current secret version instead of reusing a revoked one.

## What you will learn

- Map secret copies and consuming identities.
- Sequence routine and emergency rotation differently.
- Prove revocation at the authority and recovery after restore.
- Run a leaked-secret response in the right order.

Audience: Developers and platform engineers who store, rotate or revoke application credentials, and responders handling a leaked secret.

Before you start:

- Identify credential consumers and the authority granted by a secret.

Time: About 26 minutes: 7 to read and 19 for the exercise

## A secret is a capability with consumers and a cutoff

A database password, API token, signing key, and encryption key do different jobs. A stolen database password authorizes access while the server accepts it. A signing key may enable new forgeries. An encryption key may be needed to read years of stored data. Record purpose, owner, issuer, consumers, permissions, storage locations, refresh behavior, and the consequence of compromise before choosing a rotation mechanism.

For a billing API token, draw the path: provider issues token; deployment writes a versioned secret; workers fetch it; each worker caches it; outbound requests carry it; provider verifies it. Copies may also appear in process environments, crash reports, build logs, support exports, and backups. A secret manager governs one part of this path; it does not automatically revoke the provider credential or erase those copies.

Prefer a workload identity and short-lived issued credentials where the target service supports them. That reduces the value and lifetime of copied credentials, but creates dependencies on the issuer, clock, and refresh process. A static integration token may still be necessary; give it the smallest practical scope and an explicit owner rather than treating its existence as a design failure.

## Separate identity, access, and accidental disclosure

The workload needs permission to retrieve only its own secret versions. Granting every application access to the whole store turns one runtime compromise into a fleet-wide compromise. Likewise, a deployment identity that can update secrets should not automatically have permission to export unrelated production values.

Pass credentials through a channel appropriate to the runtime and threat model. Environment variables are convenient but may be inherited by subprocesses or included in diagnostics; files need access controls and cleanup; in-memory values can still appear in dumps. Do not promise absolute secrecy from an attacker who fully controls the process that must use the credential.

Logs should contain secret identifiers, version numbers, and outcomes, not values. Redaction must apply before storage and export, including error paths. Pattern-based scanning finds some formats and historical mistakes, but an unrecognized provider token or transformed credential can evade it. A clean scan is useful evidence about the scanner’s coverage, not a certificate that no secret exists.

How the value reaches the process matters as much as where it is stored. On Kubernetes and other runtimes, prefer mounting from a secret manager or issuing short-lived credentials to copying values into images, Git or long-lived environment variables. The runtime guide at /domains/identity-crypto/secrets/secrets-in-kubernetes-and-runtime/ covers Vault and its Agent Injector, Kubernetes Secrets with KMS v2 encryption, the Secrets Store CSI Driver, External Secrets Operator, SOPS and cloud secret-manager rotation.

## How should you generate, store and compare secrets and API keys?

Generate every secret, token and key with the operating system’s cryptographically secure random number generator, never with a general-purpose random function, a timestamp or a UUID. Store what you issue as a hash where you only need to check it later, as with API keys: a fast hash such as SHA-256 is enough for a 256-bit random value, unlike passwords, which people choose and which need a slow password hash.

This Python example issues API keys with a recognizable prefix, so users and secret scanners can spot a leaked one, a public key ID for lookup and revocation, and a 256-bit secret that is shown once and stored only as a hash. Verification compares hashes in constant time. It was tested on Python 3.14, including revoked, unknown and altered keys, and secrets that contain underscores.

```python
import hashlib
import hmac
import secrets

PREFIX = "acme_live_"  # a fixed prefix lets users and secret scanners recognize a leaked key


def new_api_key() -> tuple[str, str, str]:
    """Return (key shown once to the user, key ID to store, SHA-256 of the secret to store)."""
    key_id = secrets.token_hex(8)  # public: for lookup, logs and revocation
    secret = secrets.token_urlsafe(32)  # 256 bits from the operating system CSPRNG
    return f"{PREFIX}{key_id}_{secret}", key_id, hashlib.sha256(secret.encode()).hexdigest()


def verify_api_key(presented: str, stored_hash_for) -> str | None:
    """Return the key ID if the presented key is valid. A fast hash is enough for a 256-bit random
    secret; passwords, which people choose, need a slow password hash instead."""
    if not presented.startswith(PREFIX):
        return None
    key_id, _, secret = presented[len(PREFIX):].partition("_")
    stored = stored_hash_for(key_id)  # None if unknown or revoked
    candidate = hashlib.sha256(secret.encode()).hexdigest()
    if stored is None or not hmac.compare_digest(candidate, stored):  # constant-time comparison
        return None
    return key_id

```

| Runtime | Random bytes | Constant-time comparison |
| --- | --- | --- |
| Python | `secrets.token_bytes(32)` or `secrets.token_urlsafe(32)` | `hmac.compare_digest(a, b)` |
| Node.js | `crypto.randomBytes(32)` | `crypto.timingSafeEqual(a, b)` |
| Java | `new SecureRandom().nextBytes(bytes)` | `MessageDigest.isEqual(a, b)` |
| .NET | `RandomNumberGenerator.GetBytes(32)` | `CryptographicOperations.FixedTimeEquals(a, b)` |
| Browser | `crypto.getRandomValues(new Uint8Array(32))` | Compare on the server instead |

## Rotate the consumer graph in a deliberate order

For a provider that supports overlapping tokens, issue B while A remains valid, store B as a new version, direct new deployments and refreshes to B, and observe successful use by every consumer. Then revoke A at the provider and test it. The overlap absorbs deployment and cache delays, but is also a period when both stolen values could work.

For a provider that supports only one password, changing the issuer first can break every cached consumer. Coordinate a maintenance window, support an atomic credential change where available, or redesign the connection mechanism. Do not invent a two-key overlap in documentation when the provider cannot implement it.

Emergency revocation changes the priority: a confirmed compromised credential may need immediate invalidation despite temporary failures. Routine rotation optimizes continuity; incident response limits ongoing access. In either case, define how retries behave so a failure does not print the secret or trigger an unbounded fallback to the old version.

Encryption-key rotation needs a separate read/migrate/retire plan because old ciphertext may require the old key. Never apply a token-revocation checklist directly to data-encryption keys. Also test a restored backup: an old worker image may restart with a revoked token and must refresh safely rather than silently recreating the old configuration.

## What to do when a secret leaks

Treat a secret that reached a public repository, a shared log, a ticket or a chat as compromised from the moment it was exposed, even if it was deleted a minute later. The runbook at /domains/identity-crypto/secrets/leaked-secret-response/ gives the provider-specific steps; work in this order:

- Revoke or rotate the credential at its issuer first, using the emergency path from the rotation plan.
- Read the issuer’s audit or usage logs from the first exposure time onward, and contain any use you cannot explain.
- Find every copy: forks and clones, CI logs, build artifacts, container images, caches, tickets and chat. Deleting the file or rewriting history leaves those copies, so history rewriting is cleanup after revocation, not the remedy.
- Issue a narrowly scoped replacement through the normal rotation path, recording version identifiers rather than values.
- Remove the cause: add the commit-time checks below, and move the consumer to workload identity or short-lived credentials where the target supports them.

## Stop secrets before they are committed

GitHub push protection for users is on by default and blocks pushes of supported secret types to public repositories. Push protection for a repository’s own pushes is off by default and needs GitHub Secret Protection; turn it on and restrict who may bypass a block. Add a local pre-commit hook and a CI job that scans the full history: `gitleaks git .` exits with status 1 when a secret exists anywhere in the history, even after the file was deleted (tested with gitleaks 8.30.1), and `trufflehog git file://. --results=verified` reports the credentials it could confirm are live.

Scanners recognize known formats and high-entropy strings. An unrecognized provider token or a transformed credential can evade them, so a clean scan is evidence about the scanner’s coverage, not a certificate that no secret exists.

## Measure successful migration and explicit old-value rejection

A rotation is complete when intended consumers use the replacement and the retired credential is rejected by the authority that checks it. Updating a secret-manager alias proves neither fact. Use synthetic requests with bounded permissions and record version identifiers only.

Ordinary requests using B must succeed; requests using A after revocation must fail; an instance whose cache exceeds the rollout window must recover or fail closed according to the design. Check the same boundary for scheduled jobs, rarely used recovery paths, and a restored deployment. Set alerts for attempted use of revoked version IDs without collecting credential values.

## Guided practice

You are rotating an API token for a provider that allows two active tokens, A (old) and B (new). Write the gate that decides when A may be revoked, then the probes that prove the rotation. Produce the code and commands before reading the worked answers.

### Supplied material

```text
Secret alias switched from A to B at 09:02. Consumers fetch the alias at start and cache it for at most 10 minutes.
Inventory reported at 09:08 (consumer, version in use, last fetch, running):
  api-1, B, 09:05, yes
  api-2, B, 09:06, yes
  api-3, B, 09:07, yes
  worker-4, A, 09:01, yes
  nightly-export (CronJob), A, 01:00, no (next run 01:00 tomorrow; fetches at start)
  billing-sync, unknown, never reported, yes
A restored copy of last week's image would start with A in its environment.
Provider endpoint for checks: GET https://api.provider.example/v1/whoami with Authorization: Bearer <token>.
```

### 1. Write the revocation gate

Complete the function so it returns true only when A can be revoked without breaking a running consumer.

```python
from datetime import datetime, timedelta


def ready_to_revoke(consumers: list[dict], switched_at: datetime, cache_ttl: timedelta, now: datetime) -> bool:
    """consumers: [{"name": str, "version": "A" | "B" | None, "last_fetch": datetime | None, "running": bool}]"""
    ...

```

Worked answer: Return false if `now < switched_at + cache_ttl`; return false if any running consumer reports a version other than "B", including `None` for one that never reported; consumers that are not running do not block, because they fetch at start. In code: `if now < switched_at + cache_ttl: return False` then `return all(c["version"] == "B" for c in consumers if c["running"])`. On the 09:08 inventory it returns false twice over: `worker-4` still reports A, and `billing-sync` is unknown.

### 2. Decide the earliest revocation

Use the cache bound and the inventory to say when A can go, and what you must find out first.

Worked answer: Not before 09:12, when the 10-minute cache bound after the 09:02 switch has passed, and only once a fresh inventory shows `worker-4` on B and `billing-sync` identified and on B. A consumer that still reports A after its cache bound is not refreshing and needs fixing before revocation, not after. The nightly export will fetch B at its next start and does not block.

### 3. Write the probes

Write the commands that prove B works and A is rejected after revocation, without printing either token.

```bash
probe() { curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $1" https://api.provider.example/v1/whoami; }

```

Worked answer: With the tokens read from the secret manager into variables, never typed on the command line or echoed: `probe "$TOKEN_B"` must print `200` and, after revocation, `probe "$TOKEN_A"` must print `401` (or the provider's rejection code). Record the time, the version IDs and the status codes, not the values.

### 4. Close the restore gap

Decide what the restored image must do when it starts with A in its environment.

Worked answer: It must fetch the current version from the secret manager at start instead of trusting the value baked into its environment; if it still uses A, the first call fails with `401` and the code must refetch once and retry, never loop or fall back to A. Better, stop putting the value in the image environment at all and mount or fetch it at runtime.

Deliverable: The `ready_to_revoke` function with three unit tests (the 09:08 inventory, the same inventory at 09:13 with everything on B, and an unknown consumer), the probe commands, and a one-line rule for restored images.

Scope: The provider, consumers and times are invented; adapt the gate to how your consumers actually report the version they use.

## Executable practice

- [Insecure randomness lab: Generate unguessable API keys](https://softwaresecurity.com/labs/insecure-randomness/)

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html)
- [AWS Secrets Manager: rotation lifecycle](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html)
- [GitHub: secret scanning coverage](https://docs.github.com/en/code-security/concepts/secret-security/secret-scanning)
- [GitHub: about push protection](https://docs.github.com/en/code-security/concepts/secret-security/push-protection)
- [Gitleaks: usage and pre-commit hook](https://github.com/gitleaks/gitleaks)
- [OWASP ASVS 5.0: V13 Configuration, secret management](https://github.com/OWASP/ASVS/blob/v5.0.0_release/5.0/en/0x22-V13-Configuration.md). Version 5.0.0 (May 2025).