--- key: "case-study-circleci-2023" title: "Case study: the CircleCI security incident (January 2023)" url: "https://softwaresecurity.com/domains/runtime/cloud-posture/case-study-circleci-2023/" api: "https://softwaresecurity.com/api/v1/articles/case-study-circleci-2023.json" domain: "runtime" topic: "Runtime, cloud \u0026 APIs" type: "Case study" identifiers: ["CWE-294","CWE-522","CWE-798","A07:2025","A04:2025","V13.2.1","V13.3.1"] tags: ["Account takeover","Authentication and sessions","OAuth and OpenID Connect","Secrets management","CI/CD security","Cloud security","Incident response"] stages: ["release","operate","respond"] roles: ["platform-sre"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Case study: the CircleCI security incident (January 2023) How a session stolen from one engineer's laptop reached CircleCI's production systems and customers' stored secrets in 2023, and what platform operators and CI users should change. Canonical URL: https://softwaresecurity.com/domains/runtime/cloud-posture/case-study-circleci-2023/ ## In short In December 2022 malware on a CircleCI engineer's laptop, undetected by antivirus, stole a valid 2FA-backed single sign-on session. The attacker used it to impersonate the engineer, whose role let them generate production access tokens, and on 22 December exfiltrated customers' environment variables, tokens and keys. The data was encrypted at rest, but the attacker extracted the encryption keys from a running process. CircleCI disclosed on 4 January 2023 and told every customer to rotate every secret stored on the platform. For operators, the lesson is that a stolen session must not equal production access; for CI users, it is to store as few long-lived secrets in CI as possible and use short-lived OIDC credentials instead. - MFA protected the sign-in, not the session: once the session cookie was stolen, 2FA no longer stood in the way. - One engineer's routine privilege, generating production access tokens, was enough to reach customer secret stores. - Encryption at rest did not help, because the keys were taken from a running process that could decrypt. - Customers could not tell whether their secrets had been used elsewhere; CircleCI told everyone to rotate everything and to investigate from 16 December 2022. - CircleCI's changes: step-up authentication for production, fewer people with access, more ephemeral permissions, OAuth token rotation and a move to GitHub Apps, and audit logs for all plans. - As a CI user, prefer OIDC federation (`CIRCLE_OIDC_TOKEN_V2` on CircleCI) so a provider breach leaks nothing that outlives the job. ## Incident at a glance - Date: 2022-12-16 to 2023-01-07 - Category: Credential theft - Organization: CircleCI - Impact: A session stolen from an engineer's laptop gave production access; customers' environment variables, tokens and keys were exfiltrated and decrypted with keys taken from a running process. Every customer was told to rotate every stored secret. - How it was found: A customer reported suspicious GitHub OAuth activity on 29 December 2022; CircleCI's investigation traced it to the engineer's laptop by 4 January 2023. ### Timeline 1. 2022-12-16: Malware on an engineer's laptop steals a 2FA-backed SSO session. 2. 2022-12-19: The attacker begins reconnaissance in production systems. 3. 2022-12-22: Customer environment variables, tokens and keys are exfiltrated. 4. 2022-12-29: A customer reports suspicious GitHub OAuth activity. 5. 2023-01-04: CircleCI cuts production access, rotates hosts and discloses; customers are told to rotate all secrets. 6. 2023-01-07: Rotation of customers' GitHub OAuth tokens completes. 7. 2023-01-13: CircleCI publishes its incident report. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Entry | Malware on an engineer's laptop stole a 2FA-backed SSO session | Laptop compromised 16 December 2022; not detected by antivirus. | CircleCI incident report | | Escalation | The engineer could generate production access tokens | Used to reach a subset of databases and stores. | CircleCI incident report | | Data taken | Customer environment variables, tokens and keys | Exfiltrated 22 December 2022, after reconnaissance on 19 December. | CircleCI incident report | | Encryption at rest | Bypassed | Keys were extracted from a running process. | CircleCI incident report | | Customer action | Rotate every secret stored on CircleCI; investigate from 16 December 2022 until rotation | Fewer than five customers had reported unauthorized access to third-party systems when the report was published. | CircleCI incident report | | CircleCI's recommendation now | OIDC tokens instead of stored long-lived credentials; contexts; IP ranges | Job tokens: `CIRCLE_OIDC_TOKEN` and `CIRCLE_OIDC_TOKEN_V2`, valid one hour. | CircleCI report; CircleCI docs | ## Verify it 1. List every secret stored in your CI provider (project variables, contexts, SSH and deploy keys, API tokens) with what it can reach, and mark which could be replaced by OIDC federation. 2. Confirm cloud deploys from CI use short-lived credentials from an OIDC trust that is restricted to your organization, project and branch. 3. If you run a platform: check whether a stolen employee session could mint production access without a fresh, phishing-resistant authentication step. 4. Check where your service decrypts customer secrets, which processes hold the keys, and who can reach those processes. 5. Rehearse 'the CI provider was breached': how long would it take to revoke and replace every secret, and who would do it? ## What you will learn - Trace how a stolen session on one laptop reached customers' secrets despite 2FA and encryption at rest. - Design production access that a stolen session cannot use on its own. - Replace stored cloud keys in CI with OIDC federation restricted to a project and branch. - Plan and order a full rotation after a CI provider breach. Audience: Engineers who operate SaaS platforms with privileged production access, and teams that store secrets in hosted CI/CD Before you start: - Know how SSO sessions and session cookies work after a 2FA sign-in. - Know how CI secrets are stored and injected into jobs. - For the code: Python 3 for the rotation plan; a CircleCI project and an AWS account to try the OIDC configuration. Time: About 26 minutes: 10 to read and 16 for the exercise ## One laptop, one session, every customer's secrets CircleCI is a hosted CI/CD service. Its customers store the credentials their pipelines need, such as cloud keys, package-registry tokens and deploy keys, as project environment variables and contexts on the platform. On 4 January 2023 CircleCI told every customer to rotate all of them. Its incident report, published on 12 January (13 January in UTC), explains why. 'An unauthorized third party leveraged malware deployed to a CircleCI engineer's laptop in order to steal a valid, 2FA-backed SSO session. This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software.' The malware performed 'session cookie theft, enabling them to impersonate the targeted employee in a remote location and then escalate access to a subset of our production systems.' Because the employee 'had privileges to generate production access tokens as part of the employee's regular duties', the attacker could 'access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys.' The data was encrypted at rest. That did not matter: 'the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data.' CircleCI's report is the only primary source for the intrusion; it does not name the attacker. ## Timeline | When (UTC) | Event | Source | | --- | --- | --- | | 2022-12-16 | Malware compromises a CircleCI engineer's laptop and steals a 2FA-backed SSO session. | CircleCI report | | 2022-12-19 | The attacker begins reconnaissance in production. | CircleCI report | | 2022-12-22 | Customer data is exfiltrated; CircleCI's last record of unauthorized activity in production. | CircleCI report | | 2022-12-29 | A customer alerts CircleCI to suspicious GitHub OAuth activity. | CircleCI report | | 2022-12-31 | CircleCI starts rotating all customers' GitHub OAuth tokens on their behalf (completed 7 January). | CircleCI report | | 2023-01-04 | CircleCI cuts off the employee, then production access for nearly all employees, and rotates all production hosts by 22:30. | CircleCI report | | 2023-01-05 | 02:30: disclosure to customers, asking them to rotate every secret; 03:26: all Project API Tokens revoked. | CircleCI report | | 2023-01-06 | Personal API Tokens created before 5 January revoked; Bitbucket tokens rotated with Atlassian. | CircleCI report | | 2023-01-07 to 12 | CircleCI and AWS notify customers of potentially affected AWS tokens. | CircleCI report | | 2023-01-13 | CircleCI publishes its incident report (12 January, US Pacific time). | CircleCI report | ## Why 2FA and encryption did not stop it Two-factor authentication protects the moment of sign-in. After that, the session cookie is the credential, and malware on the endpoint can copy it. The same pattern ran through the Okta support-system breach later in 2023 (/domains/identity-crypto/authentication/case-study-okta-har-2023/), and the session guide (/domains/identity-crypto/authentication/session-lifecycle/) covers what limits a stolen session: short lifetimes, binding to the device, and re-authentication before sensitive actions. The decisive step was the role. The engineer could generate production access tokens as a normal part of the job, so a stolen session became production access with no further check. CircleCI's response attacked exactly that: production access was cut back to 'a very limited number of employees', and 'For the employees who retain production access, we have added additional step-up authentication steps and controls ... even in the case of a stolen 2FA-backed SSO session.' Its plan added 'reducing session trust, adding additional authentication factors, and performing more regular access rotations' and 'making our system permissions more ephemeral'. Encryption at rest protects disks and backups, not data that a running service decrypts. If a process can decrypt customer secrets, anyone who controls that process can too. Limit which services ever hold decryption keys, keep that step in a separate, tightly accessed component, and monitor it. The envelope encryption guide (/domains/identity-crypto/key-management-pki/envelope-encryption-rotation/) covers key separation. ## What it meant for customers: rotate everything From the customer's side, the platform that held their secrets had been breached and they could not know which secrets had been used. CircleCI was explicit: 'If you stored secrets on our platform during this time period, assume they have been accessed', and 'there is no way for us to know if your secrets were used for unauthorized access to those third-party systems.' It advised investigating activity from 16 December 2022 until each secret was rotated, and published IP addresses, VPN providers, file names and a domain to search for. When the report was published, 'fewer than 5 customers' had reported unauthorized access to third-party systems. CircleCI built tooling to help: a script to find secrets for rotation, an `updated_at` field in the Contexts API to confirm rotation, and audit logs made available to customers on all plans. The rotation still fell on every customer, for every secret, at once. The leaked secret guide (/domains/identity-crypto/secrets/leaked-secret-response/) covers how to run a rotation like that; the plan script below orders it. The best position to be in was to have stored nothing worth stealing. CircleCI's own best-practice list starts with 'Use OIDC tokens wherever possible to avoid storing long-lived credentials in CircleCI.' A job that federates to the cloud provider with a token valid for one hour leaves nothing in the CI platform that outlives the job. The workload identity guide (/domains/identity-crypto/identity/workload-identity-models/) covers the trust setup. ## Deploy with OIDC instead of stored cloud keys This CircleCI configuration uses the `circleci/aws-cli` orb, pinned to 5.4.2, the current release, to exchange the job's `CIRCLE_OIDC_TOKEN_V2` for AWS credentials through `sts:AssumeRoleWithWebIdentity`, for 15 minutes. No AWS key is stored in CircleCI. The YAML parses; it was checked against CircleCI's OIDC documentation and the orb's source, not run in a CircleCI project for this page. ```yaml # .circleci/config.yml: deploy with short-lived AWS credentials from CircleCI's # OIDC token instead of a stored AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. version: 2.1 orbs: aws-cli: circleci/aws-cli@5.4.2 jobs: deploy: executor: aws-cli/default steps: - checkout - aws-cli/setup: role_arn: arn:aws:iam::123456789012:role/circleci-deploy role_session_name: circleci-deploy session_duration: "900" # 15 minutes, the minimum AWS allows - run: aws sts get-caller-identity # proves which role the job holds workflows: release: jobs: - deploy: context: aws-deploy # restrict this context to the release group filters: branches: only: main ``` ## Restrict which jobs can assume the role The AWS role's trust policy decides who can use it. CircleCI's issuer is `https://oidc.circleci.com/org/`, the default audience is the organization ID, and the V2 `sub` claim names the organization, project, user, VCS origin and ref. The policy below accepts tokens only for one project's `main` branch. Replace `ORG_ID` and `PROJECT_ID` with your IDs, and add the OIDC provider to IAM first. Without the `sub` condition, any project in your CircleCI organization could deploy. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.circleci.com/org/ORG_ID" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": {"oidc.circleci.com/org/ORG_ID:aud": "ORG_ID"}, "StringLike": { "oidc.circleci.com/org/ORG_ID:sub": "org/ORG_ID/project/PROJECT_ID/user/*/vcs-origin/*/vcs-ref/refs/heads/main" } } } ] } ``` ## Order the rotation when your CI provider is breached The script reads an inventory of the secrets stored in your CI platform and prints the order to act in: production first, then by kind (cloud and signing keys before others) and breadth of scope. Secrets that can be replaced by OIDC are revoked rather than reissued. It ran with Python 3.14 on a five-secret sample. Keep the inventory current before you need it; building it during an incident is the slow part. ```python """Plan secret rotation after a CI provider breach: assume every stored secret leaked. Input: a JSON inventory of the secrets stored in the CI platform, one entry per secret: name, kind (cloud-key, deploy-key, api-token, signing-key, oauth...), scope (what it can reach), prod (true/false), oidc_possible (true/false). Output: the order to act in, and which secrets to replace with OIDC instead of reissuing. """ import json import sys KIND_WEIGHT = {"cloud-key": 3, "signing-key": 3, "deploy-key": 2, "api-token": 2, "oauth": 2} def plan(secrets): def priority(s): return (s.get("prod", False), KIND_WEIGHT.get(s["kind"], 1), len(s.get("scope", []))) for s in sorted(secrets, key=priority, reverse=True): action = "revoke now; replace with OIDC" if s.get("oidc_possible") else "rotate now" yield f"{s['name']:<24} {s['kind']:<12} {'prod' if s.get('prod') else 'non-prod':<9} {action}" if __name__ == "__main__": with open(sys.argv[1], encoding="utf-8") as f: for line in plan(json.load(f)): print(line) ``` ## Which controls would have stopped or limited it? | Control | Who | SSDF practice or standard | Guide | | --- | --- | --- | --- | | Endpoint detection and hardening for engineers with production access | Platform operator | SSDF PO.5.2 | /domains/supply-chain/ci-cd/developer-workstation-security/ | | Step-up, phishing-resistant authentication before minting production access; short, bound sessions | Platform operator | SSDF PO.5.1; ASVS V7 | /domains/identity-crypto/authentication/session-lifecycle/ | | Few people with standing production access; just-in-time, expiring grants | Platform operator | SSDF PO.5.1 | /domains/runtime/cloud-posture/cloud-iam-trust/ | | Decrypt customer secrets only in an isolated component whose keys no general process holds | Platform operator | SSDF PS.1.1; ASVS V13.3.1 | /domains/identity-crypto/key-management-pki/envelope-encryption-rotation/ | | Audit logs for every customer, so they can investigate the provider | Platform operator | ASVS V16.2.1 | /domains/runtime/detection/security-event-logging/ | | Short-lived OIDC credentials instead of stored cloud keys; scoped contexts | CI user | ASVS V13.2.1 | /domains/identity-crypto/identity/workload-identity-models/ | | An inventory of CI secrets and a rehearsed rotation | CI user | SSDF RV.2.2 | /domains/identity-crypto/secrets/leaked-secret-response/ | ## Lessons that still apply - A session is a credential. Protect what it can do: step-up authentication for production, short lifetimes, device binding. - Standing production access for routine work turns one stolen session into a breach. Make access just-in-time and expiring. - Encryption at rest does not protect data from a process that can decrypt it. - As a CI user, assume your provider can be breached; store as little as possible there and federate the rest. - Treat it as a systems failure, as CircleCI did: 'a security incident is a systems failure', not one employee's mistake. ## What the public evidence does not establish CircleCI's report does not say how the malware reached the engineer's laptop, who was behind the attack, or how many customers' secrets were read or later used; at publication, fewer than five customers had reported unauthorized access to third-party systems, which is a lower bound. CircleCI engaged third-party investigators to validate its findings, but their report is not public. The timeline and mechanism on this page come from CircleCI alone; no regulator or independent review has published findings on the incident. ## Guided practice It is 5 January 2023 and your hosted CI provider has just told every customer to rotate all secrets stored on it. Decide what to do with each item. Treat the facts as true. ### Supplied material ```text E1: AWS access key for a deploy user with AdministratorAccess, stored as a project variable since 2020. E2: npm publish token for your public packages, in an org-wide context. E3: GitHub deploy key with write access to the infrastructure repo. E4: Slack webhook for build notifications. E5: Nothing else is logged: you have CloudTrail for AWS, npm and GitHub audit logs, and no record of when E1 was last used outside CI. ``` ### 1. Order the work In what order do you act, and why? Worked answer: E1, then E2 and E3, then E4. E1 can change your whole AWS account; E2 can ship code to your users; E3 can change infrastructure. Revoke before you rotate where a replacement is not needed immediately. ### 2. Replace E1 What replaces E1? Worked answer: An IAM role assumed through the CI provider's OIDC, restricted to your organization, project and main branch, with only the permissions the deploy needs; then delete the access key. ### 3. Investigate What do you search, and over what window? Worked answer: CloudTrail for E1's key, npm publish history for E2, and GitHub audit logs for E3, from 16 December 2022 until each secret was revoked, including the provider's published IP addresses. ### 4. Prevent a repeat What changes so the next provider breach is a non-event? Worked answer: Short-lived federated credentials for cloud and npm publishing, deploy keys replaced by app installations with narrow scopes, and a maintained inventory of every secret in CI with a rotation runbook. Deliverable: An ordered response list with the replacement for each secret, the logs you checked and the window, and two changes to how secrets live in CI. Scope: The fixture is fictional. Whether npm or another registry accepts OIDC-based publishing from a given CI provider depends on that registry's support; check it before relying on it. Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [CircleCI: Jan 4, 2023 security incident report](https://circleci.com/blog/jan-4-2023-incident-report/). 12 January 2023 (US Pacific). - [CircleCI docs: Using OpenID Connect tokens in jobs](https://circleci.com/docs/guides/permissions-authentication/openid-connect-tokens/). Updated 28 September 2026. - [CircleCI-Public/aws-cli-orb: setup command and web identity example](https://github.com/CircleCI-Public/aws-cli-orb). v5.4.2 (15 June 2026). - [AWS: Create a role for OpenID Connect federation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_create_for-idp_oidc.html) - [OWASP ASVS 5.0.0: V13 Configuration](https://github.com/OWASP/ASVS/blob/v5.0.0_release/5.0/en/0x22-V13-Configuration.md). Version 5.0.0.