Case study: the CircleCI security incident (January 2023)

CircleCI is a hosted CI/CD service. Its customers store the credentials their pipelines need, such as cloud keys, package-registry tokens and deploy keys, as project environment variables and contexts on the platform. On 4 January 2023 CircleCI told every customer to rotate all of them. Its incident report, published on 12 January (13 January in UTC), explains why.

'An unauthorized third party leveraged malware deployed to a CircleCI engineer's laptop in order to steal a valid, 2FA-backed SSO session. This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software.' The malware performed 'session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.' Because the employee 'had privileges to generate production access tokens as part of the employee's regular duties', the attacker could 'access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys.'

The data was encrypted at rest. That did not matter: 'the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.' CircleCI's report is the only primary source for the intrusion; it does not name the attacker.

Two-factor authentication protects the moment of sign-in. After that, the session cookie is the credential, and malware on the endpoint can copy it. The same pattern ran through the Okta support-system breach later in 2023 (/domains/identity-crypto/authentication/case-study-okta-har-2023/), and the session guide (/domains/identity-crypto/authentication/session-lifecycle/) covers what limits a stolen session: short lifetimes, binding to the device, and re-authentication before sensitive actions.

The decisive step was the role. The engineer could generate production access tokens as a normal part of the job, so a stolen session became production access with no further check. CircleCI's response attacked exactly that: production access was cut back to 'a very limited number of employees', and 'For the employees who retain production access, we have added additional step-up authentication steps and controls ... even in the case of a stolen 2FA-backed SSO session.' Its plan added 'reducing session trust, adding additional authentication factors, and performing more regular access rotations' and 'making our system permissions more ephemeral'.

Encryption at rest protects disks and backups, not data that a running service decrypts. If a process can decrypt customer secrets, anyone who controls that process can too. Limit which services ever hold decryption keys, keep that step in a separate, tightly accessed component, and monitor it. The envelope encryption guide (/domains/identity-crypto/key-management-pki/envelope-encryption-rotation/) covers key separation.

From the customer's side, the platform that held their secrets had been breached and they could not know which secrets had been used. CircleCI was explicit: 'If you stored secrets on our platform during this time period, assume they have been accessed', and 'there is no way for us to know if your secrets were used for unauthorized access to those third-party systems.' It advised investigating activity from 16 December 2022 until each secret was rotated, and published IP addresses, VPN providers, file names and a domain to search for. When the report was published, 'fewer than 5 customers' had reported unauthorized access to third-party systems.

CircleCI built tooling to help: a script to find secrets for rotation, an updated_at field in the Contexts API to confirm rotation, and audit logs made available to customers on all plans. The rotation still fell on every customer, for every secret, at once. The leaked secret guide (/domains/identity-crypto/secrets/leaked-secret-response/) covers how to run a rotation like that; the plan script below orders it.

The best position to be in was to have stored nothing worth stealing. CircleCI's own best-practice list starts with 'Use OIDC tokens wherever possible to avoid storing long-lived credentials in CircleCI.' A job that federates to the cloud provider with a token valid for one hour leaves nothing in the CI platform that outlives the job. The workload identity guide (/domains/identity-crypto/identity/workload-identity-models/) covers the trust setup.

This CircleCI configuration uses the circleci/aws-cli orb, pinned to 5.4.2, the current release, to exchange the job's CIRCLE_OIDC_TOKEN_V2 for AWS credentials through sts:AssumeRoleWithWebIdentity, for 15 minutes. No AWS key is stored in CircleCI. The YAML parses; it was checked against CircleCI's OIDC documentation and the orb's source, not run in a CircleCI project for this page.

# .circleci/config.yml: deploy with short-lived AWS credentials from CircleCI's
# OIDC token instead of a stored AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
version: 2.1

orbs:
  aws-cli: circleci/aws-cli@5.4.2

jobs:
  deploy:
    executor: aws-cli/default
    steps:
      - checkout
      - aws-cli/setup:
          role_arn: arn:aws:iam::123456789012:role/circleci-deploy
          role_session_name: circleci-deploy
          session_duration: "900" # 15 minutes, the minimum AWS allows
      - run: aws sts get-caller-identity # proves which role the job holds

workflows:
  release:
    jobs:
      - deploy:
          context: aws-deploy # restrict this context to the release group
          filters:
            branches:
              only: main

The AWS role's trust policy decides who can use it. CircleCI's issuer is https://oidc.circleci.com/org/<organization_id>, the default audience is the organization ID, and the V2 sub claim names the organization, project, user, VCS origin and ref. The policy below accepts tokens only for one project's main branch. Replace ORG_ID and PROJECT_ID with your IDs, and add the OIDC provider to IAM first. Without the sub condition, any project in your CircleCI organization could deploy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.circleci.com/org/ORG_ID"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {"oidc.circleci.com/org/ORG_ID:aud": "ORG_ID"},
        "StringLike": {
          "oidc.circleci.com/org/ORG_ID:sub": "org/ORG_ID/project/PROJECT_ID/user/*/vcs-origin/*/vcs-ref/refs/heads/main"
        }
      }
    }
  ]
}

The script reads an inventory of the secrets stored in your CI platform and prints the order to act in: production first, then by kind (cloud and signing keys before others) and breadth of scope. Secrets that can be replaced by OIDC are revoked rather than reissued. It ran with Python 3.14 on a five-secret sample. Keep the inventory current before you need it; building it during an incident is the slow part.

"""Plan secret rotation after a CI provider breach: assume every stored secret leaked.

Input: a JSON inventory of the secrets stored in the CI platform, one entry per
secret: name, kind (cloud-key, deploy-key, api-token, signing-key, oauth...),
scope (what it can reach), prod (true/false), oidc_possible (true/false).
Output: the order to act in, and which secrets to replace with OIDC instead of reissuing.
"""
import json
import sys

KIND_WEIGHT = {"cloud-key": 3, "signing-key": 3, "deploy-key": 2, "api-token": 2, "oauth": 2}


def plan(secrets):
    def priority(s):
        return (s.get("prod", False), KIND_WEIGHT.get(s["kind"], 1), len(s.get("scope", [])))

    for s in sorted(secrets, key=priority, reverse=True):
        action = "revoke now; replace with OIDC" if s.get("oidc_possible") else "rotate now"
        yield f"{s['name']:<24} {s['kind']:<12} {'prod' if s.get('prod') else 'non-prod':<9} {action}"


if __name__ == "__main__":
    with open(sys.argv[1], encoding="utf-8") as f:
        for line in plan(json.load(f)):
            print(line)

• A session is a credential. Protect what it can do: step-up authentication for production, short lifetimes, device binding.
• Standing production access for routine work turns one stolen session into a breach. Make access just-in-time and expiring.
• Encryption at rest does not protect data from a process that can decrypt it.
• As a CI user, assume your provider can be breached; store as little as possible there and federate the rest.
• Treat it as a systems failure, as CircleCI did: 'a security incident is a systems failure', not one employee's mistake.

CircleCI's report does not say how the malware reached the engineer's laptop, who was behind the attack, or how many customers' secrets were read or later used; at publication, fewer than five customers had reported unauthorized access to third-party systems, which is a lower bound. CircleCI engaged third-party investigators to validate its findings, but their report is not public. The timeline and mechanism on this page come from CircleCI alone; no regulator or independent review has published findings on the incident.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. CircleCI: Jan 4, 2023 security incident report. 12 January 2023 (US Pacific).
2. CircleCI docs: Using OpenID Connect tokens in jobs. Updated 28 September 2026.
3. CircleCI-Public/aws-cli-orb: setup command and web identity example. v5.4.2 (15 June 2026).
4. AWS: Create a role for OpenID Connect federation
5. OWASP ASVS 5.0.0: V13 Configuration. Version 5.0.0.

Reference information

Tags

Account takeoverAuthentication and sessionsOAuth and OpenID ConnectSecrets managementCI/CD securityCloud securityIncident response

SDLC phase

ReleaseMaintenance

Role

Platform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-13.2.1 · v5.0.0-13.3.1

Download this guide as Markdown

Continue exploring