Contain a leaked secret, then investigate and clean up

Treat a secret as compromised as soon as it reaches any place other people or systems can read: a public or private repository, a CI log, a container image, a published package, a chat message, a ticket or a paste site. Attackers scan public code for keys continuously, and GitHub forwards tokens it finds in public repositories to the providers that issued them. Deleting the file or the commit removes your copy, not anyone else's.

Only the issuer can make the value useless, so revoke, deactivate or rotate it there first, using an administrator identity rather than the leaked credential. Accept a short outage if consumers break: a failed deployment costs less than an attacker holding working credentials. Then work through the rest in this order.

• Contain: revoke or deactivate the credential at the issuer.
• Cut derived access: end the sessions and tokens that were minted with it and outlive it.
• Replace: issue a new credential with a tighter scope, ideally short-lived, and deploy it to legitimate consumers.
• Assess: read the provider's audit logs from the start of the exposure window, looking for use and persistence.
• Clean up: remove the value from code, history, logs, images and packages, and fix the path that leaked it.
• Record: keep a timeline of exposure, detection, revocation and the last suspicious event.

Identify the issuer, the account or project, the identity behind the secret and its permissions. Note whether it can mint other credentials: an AWS access key can call sts:GetSessionToken, a Google Cloud service account key can request access tokens, and a GitHub App private key can create installation tokens. A read-only analytics key and an organization administrator key need very different follow-up.

Set the exposure window from evidence. It starts when the value first became readable by someone else: for git, the push time in the hosting provider's activity or audit log, not the commit dates, which the committer can set to anything; for a public repository, also the moment the repository or a fork became public. It ends when the issuer rejects the value. Everything inside that window is in scope.

Look for copies beyond the one in the alert. The same value often sits in an .env file in another repository, a CI variable printed by a debug step, a container image layer, a compiled front-end or mobile bundle, crash reports and old forks. Scan them with the same tool and treat every hit as part of the same incident. When the copy is a file the web server serves, such as /.env or a .git folder, also stop serving it (/domains/appsec/secure-coding/information-disclosure-misconfiguration/).

When the leak is at a provider that stores your secrets, the scope is everything stored there. CircleCI's January 2023 breach meant every customer had to rotate every stored secret (/domains/runtime/cloud-posture/case-study-circleci-2023/); a SAS token published in a repository, as in Microsoft's 2023 exposure, is scoped by what the token encodes (/domains/runtime/cloud-posture/case-study-microsoft-sas-token/).

Run these with an administrator identity and current CLIs, and rehearse them before you need them. Each row lists the containing action, where use shows up, and the trap that most often leaves access open.

These commands deactivate the key, deny every session credential issued before the cutoff, and pull the key's recent history from each Region. The aws:TokenIssueTime key is present only on requests signed with temporary credentials, so the deny does not touch the user's long-term credentials; it can be removed once 36 hours have passed or when the user is deleted. Then repeat the lookup for the temporary key IDs returned by any GetSessionToken or AssumeRole call the key made.

# Run as an administrator, never with the leaked key.
USER=ci-deployer
KEY=AKIAIOSFODNN7EXAMPLE

# 1. Stop the key itself.
aws iam update-access-key --user-name "$USER" --access-key-id "$KEY" --status Inactive

# 2. Session credentials minted from it (GetSessionToken) stay valid for up to 36 hours.
#    Deny every session issued before now; long-term credentials are not affected.
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
cat > deny-old-sessions.json <<EOF
{"Version": "2012-10-17", "Statement": {"Effect": "Deny", "Action": "*", "Resource": "*",
  "Condition": {"DateLessThan": {"aws:TokenIssueTime": "$NOW"}}}}
EOF
aws iam put-user-policy --user-name "$USER" --policy-name deny-sessions-before-revocation \
  --policy-document file://deny-old-sessions.json

# 3. What did it do? Event history: 90 days of management events, one Region per call.
for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  aws cloudtrail lookup-events --region "$r" \
    --lookup-attributes AttributeKey=AccessKeyId,AttributeValue="$KEY" \
    --query 'Events[].[EventTime,EventName,Username]' --output text
done

Query by the credential's own identifier, from the start of the exposure window until after revocation, so you also see denied retries: CloudTrail by access key ID, Cloud Audit Logs by service account key name, Entra service principal sign-ins by application ID, the GitHub audit log by token hash, the provider's request log for an API key, and sshd logs by key fingerprint.

Attackers usually begin with an identity check (AWS sts:GetCallerIdentity needs no permissions and cannot be denied), then enumerate, then read data or create resources. Instances launched in Regions you never use often mean fraud that runs up charges, which is why AWS's quarantine policy blocks instance launches. The findings that matter most are footholds that survive your revocation: new users, keys, roles or trust-policy changes, OAuth applications, SSH and deploy keys, webhooks, CI secrets, scheduled jobs and functions. Remove them before you call the incident contained.

Know the gaps. CloudTrail event history keeps 90 days of management events per Region, and S3 object reads appear only if a trail recorded data events. Google Cloud Data Access audit logs are off by default except for BigQuery, and Azure Storage resource logs exist only if a diagnostic setting routed them. Where the relevant log never existed, absence of evidence proves nothing: say so in the incident record and rotate whatever the identity could read, such as secrets in a vault it could access.

# GitHub Enterprise Cloud: find audit log events made with a leaked token.
# Read the token from a file so it never lands in shell history.
HASH=$(tr -d '\n' < leaked-token.txt | openssl dgst -sha256 -binary | base64)
rm leaked-token.txt
echo "Search the organization audit log for: hashed_token:\"$HASH\""
# Same value as GitHub's documented: echo -n TOKEN | openssl dgst -sha256 -binary | base64

Rewriting history is cleanup, not containment. After a force-push the value still exists in other people's clones, in forks, in CI caches and mirrors, in pull request refs and in the hosting provider's cached views. GitHub's procedure starts by revoking the secret and notes that, once it is revoked, a rewrite may not be worth its side effects. Rewrite when the repository must not keep the value at all, for example because it is public or the secret cannot be rotated quickly.

Use git-filter-repo 2.47 or later with --sensitive-data-removal, which fetches every ref before rewriting and prints the first changed commits. Merge or close open pull requests first, lift branch protection for the force-push, and have collaborators re-clone instead of pulling: a single merge from an old clone brings the commit back. GitHub refuses pushes to refs/pull/*, so removing those refs, cached views and server-side objects takes a support request that quotes the first changed commits.

In a local test with git-filter-repo 2.47.0, a fresh clone after the force-push held no trace of a planted fake key, a teammate's clone taken before the rewrite still held it, and the server repository still returned the old commit object until garbage collection. That is why rotation stays mandatory.

# Revoke the secret first. Then rewrite in a fresh clone (git-filter-repo 2.47 or later).
git clone https://github.com/ORG/REPO.git && cd REPO

# Keep the replacements file outside the repository and delete it afterwards.
cat > ../replacements.txt <<'EOF'
AKIAIOSFODNN7EXAMPLE==>REMOVED_AWS_ACCESS_KEY_ID
regex:aws_secret_access_key = ".*"==>aws_secret_access_key = "REMOVED"
EOF

git filter-repo --sensitive-data-removal --replace-text ../replacements.txt
git log -p --all | grep -c 'AKIAIOSFODNN7EXAMPLE'   # expect 0

# Overwrites every branch and tag. GitHub rejects refs/pull/*: that part needs Support,
# quoting the "First Changed Commit(s)" that git-filter-repo printed.
git push --force --mirror origin

Scanners match known token formats and high-entropy strings, so they find most provider keys and miss many hand-made passwords. GitHub secret scanning, which runs free on public repositories and needs GitHub Secret Protection on private ones, reads the full history of every branch as well as issues, pull requests, discussions and wikis. For public repositories and public npm packages it sends partner tokens to their issuers, and GitHub revokes its own tokens pushed to public repositories or gists. Push protection for users is on by default and blocks secrets in pushes to public repositories; repository push protection, which also covers private repositories, needs GitHub Secret Protection and is off by default.

Add a local layer so a secret never leaves the laptop (the fix card at /fix/hardcoded-secrets/ covers removing secrets from code): Gitleaks or TruffleHog as a pre-commit hook, and the same tool over the full history in CI. TruffleHog verifies candidates against the provider's API by default (--no-verification turns that off); verification confirms a live secret but is itself a use of the credential and appears in the provider's logs.

Prove that the control works. With Gitleaks 8.30.1 and pre-commit 4.6.2, the published gitleaks-system hook let a commit containing an AWS-format key through: pre-commit appended the staged file names, Gitleaks treated them as the repository path, logged an error and reported no leaks. Adding pass_filenames: false, as below, made the same commit fail.

# .pre-commit-config.yaml  (needs the gitleaks binary on PATH; then run: pre-commit install)
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.30.1
    hooks:
      - id: gitleaks-system
        pass_filenames: false   # without this the hook scanned nothing and passed

# In CI, scan the full history of every branch:
#   gitleaks git --redact --log-opts="--all" .

The cheapest incident is one where the leaked value is already useless. Replace long-lived keys with credentials the platform issues for minutes or hours, and keep the static secrets that remain in a manager that rotates them and logs every read.

• CI and deployments: OIDC federation from the CI system to the cloud (for example GitHub Actions to AWS, Google Cloud or Azure) instead of stored cloud keys, with trust conditions on repository, branch and environment; /domains/identity-crypto/identity/workload-identity-models/ shows the trust policy.
• Workloads: attached roles and managed identities instead of key files; in Google Cloud, block new service account keys with the iam.managed.disableServiceAccountKeyCreation organization policy.
• API keys that must exist: restricted scopes, IP or access policies where offered, an expiry date, an owner and a rotation procedure you have actually run; for GitHub, fine-grained tokens with an organization maximum-lifetime policy.
• Databases and internal services: dynamic credentials with a lease, so revocation is an expiry rather than a hunt.
• SSH and TLS: short-lived SSH certificates or hardware-backed keys, and automated certificate renewal, which the shrinking public certificate lifetimes now require anyway.
• Hygiene: ignore rules for env and key files, no secrets in CI output or error pages, and one tested runbook per credential type.

Containment is complete when the issuer rejects the old value, every derived session has expired or been denied, footholds found in the logs are gone, and the replacement works for legitimate consumers. Record the exposure, detection, revocation and last-malicious-use times. The gap between exposure and revocation is the number to shrink next time, and it shrinks fastest with push protection in front and short-lived credentials behind.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Secrets Management Cheat Sheet: Incident Response
2. GitHub Docs: Removing sensitive data from a repository
3. GitHub Docs: Push protection
4. AWS IAM: Revoke IAM role temporary security credentials (aws:TokenIssueTime)
5. AWS CloudTrail: Viewing recent management events with the AWS CLI
6. Google Cloud IAM: Disable and enable service account keys
7. Microsoft Entra ID Protection: Securing workload identities (leaked credentials remediation)
8. Stripe Docs: Rotate an API key
9. Let's Encrypt: Revoking certificates
10. OpenSSH sshd_config(5): RevokedKeys

Reference information

Tags

Workload identitySecrets managementCloud securityIncident response

SDLC phase

DevelopmentMaintenance

Role

DevelopersAppSec engineersPlatform engineers and SREs

CAPEC attack patterns

• CAPEC-560 at MITRE
• CAPEC-37 at MITRE
• CAPEC-191 at MITRE

Download this guide as Markdown

Continue exploring