Incident response

Respond to an exploited or malicious dependency

When a dependency vulnerability is being exploited, or a package version is malicious, triage gives way to incident response: find every running copy, contain it, patch and redeploy, hunt for exploitation, rotate secrets and close on deployed evidence.

Maintenance · AppSec engineers, Platform engineers and SREs

Case study: MOVEit Transfer mass exploitation (CVE-2023-34362)

How one unauthenticated SQL injection in an internet-facing file transfer product let the Cl0p group steal data from thousands of organizations in days in 2023, how defenders found the web shell, and the vendor and customer controls that would have limited it.

Maintenance, Security program · Developers, AppSec engineers

Set up a PSIRT: services, intake, triage, SLAs and the CNA role

How to structure and run a product security incident response team on the FIRST PSIRT Services Framework: organizational models, intake channels, triage, response targets, a case record and SLA script, becoming a CVE Numbering Authority under the CNA rules 4.2.0, and coordinating with other vendors.

Maintenance · AppSec engineers, Security leaders

Respond to product security incidents: plan, tabletop and postmortem

An incident response plan for product security incidents (exploited vulnerabilities, compromised releases, leaked signing keys, malicious dependencies shipped to customers), aligned with NIST SP 800-61 Rev. 3, with reporting clocks, a two-scenario tabletop exercise template and a blameless postmortem template.

Maintenance, Security program · AppSec engineers, Platform engineers and SREs, Security leaders

Case study: First American's EaglePro document exposure (2019)

How sequential IDs in unauthenticated links exposed First American's title documents in 2019, after its own penetration test found the flaw and a clerical error let it go unfixed.

Development, Testing, Maintenance, Security program · Developers, AppSec engineers, Security leaders

Case study: CrowdStrike Channel File 291 (July 2024)

How a CrowdStrike content update crashed Windows hosts in July 2024: a field-count mismatch hidden by wildcard tests, a missing bounds check, and no staged rollout.

Development, Release, Maintenance · Developers, Platform engineers and SREs

Investigate a suspected malicious package safely

Distinguish package substitution, publisher compromise and dangerous installation behavior while containing exposure and preserving evidence.

Development, Maintenance · Platform engineers and SREs

Case study: the Shai-Hulud npm worm (2025-2026)

How a self-replicating npm worm turned stolen tokens into hundreds of infected packages in 2025, how its 2026 successors hijacked release pipelines and trusted publishing, what npm and GitHub changed in response, and the publishing, install and response practices that stop it.

Development, Release, Maintenance · Developers, AppSec engineers, Platform engineers and SREs

Case study: the tj-actions/changed-files compromise (CVE-2025-30066)

In March 2025 an attacker moved the version tags of a popular GitHub Action, and of a reviewdog action before it, to code that printed CI secrets into build logs. This case study traces the token chain back to one pull_request_target workflow, sizes the exposure, and shows how it was caught and which controls held.

Development, Release, Maintenance · Developers, AppSec engineers, Platform engineers and SREs

Create security logs someone can investigate

Record meaningful decisions with consistent structure, protect the resulting evidence, and prove that important events reach an operational owner.

Maintenance · Developers, AppSec engineers, Platform engineers and SREs

Turn runtime events into a tested investigation and containment path

Collect evidence that answers a concrete security question and rehearse the response without destroying the facts needed to investigate.

Maintenance · Platform engineers and SREs

Detect container and host threats at runtime with eBPF sensors and cloud services

Watch process, file and network activity inside containers with an eBPF sensor or a managed cloud agent, write and test rules for the attacks you expect, tune them with narrow exceptions and contain a compromised pod without destroying evidence.

Maintenance · Platform engineers and SREs

Case study: the CircleCI security incident (January 2023)

How a session stolen from one engineer's laptop reached CircleCI's production systems and customers' stored secrets in 2023, and what platform operators and CI users should change.

Release, Maintenance · Platform engineers and SREs

Case study: Microsoft AI researchers' over-permissive SAS token (2023)

How an over-permissive Azure Storage SAS token in a public repository for AI models exposed internal Microsoft data in 2023, and how to scope, scan for and revoke signed URLs.

Maintenance · Platform engineers and SREs

Secrets management: storage, rotation and leaked-secret response

Follow one application credential from issuance through caching, rotation, revocation and restored backups, prove that its old value stops working, and respond correctly when a secret leaks.

Development, Maintenance · Developers, Platform engineers and SREs

Contain a leaked secret, then investigate and clean up

Revoke an exposed API key, token or private key at its issuer before anything else, end the sessions it minted, check the provider's logs, then remove every copy and replace it with a credential that expires on its own.

Development, Maintenance · Developers, AppSec engineers, Platform engineers and SREs

Case study: the LastPass breach (2022)

How an attacker went from a compromised developer laptop to a DevOps engineer's home computer, the keys to LastPass's cloud backups and a copy of customer vaults; why each vault's safety then rested on its master password and PBKDF2 iteration count; and what it teaches about backup keys, privileged endpoints, KDF upgrades and unencrypted metadata.

Design, Development, Maintenance · AppSec engineers, Platform engineers and SREs

Case study: the Okta support-system breach and session tokens in HAR files (2023)

How a service account password saved to a personal browser profile let an attacker download customers' HAR files from Okta's support system, replay the session tokens inside them against five customers, and stay unseen for two weeks; and why HAR files are credentials, admin sessions need binding, and every exposed credential must be rotated.

Development, Maintenance · Developers, Platform engineers and SREs

Case study: certificate expiry outages at Ericsson (2018) and Microsoft Teams (2020), and Equifax's blind spot

How an expired certificate inside Ericsson's core-network software cut mobile data for millions in about 11 countries, how an expired authentication certificate took Microsoft Teams offline, how an expired inspection certificate left Equifax blind for 19 months, through its 2017 breach, and the inventory, automation and testing that prevent each.

Maintenance · Developers, AppSec engineers, Platform engineers and SREs

Case study: Storm-0558 and the forged Microsoft tokens (2023)

How a stolen 2016 consumer signing key and a token validation gap let Storm-0558 read enterprise Exchange Online mail in 2023, and what the review board found about rotation, logging and culture.

Development, Maintenance, Security program · Developers, Platform engineers and SREs

Build a runbook for CRA vulnerability and incident reporting

Since 11 September 2026, manufacturers must report actively exploited vulnerabilities and severe incidents through ENISA's Single Reporting Platform within 24 and 72 hours. Decide what is reportable, start the clock at the right moment, file through the platform and keep an evidence trail.

Maintenance, Security program · AppSec engineers, Security leaders

Contain cascading failures and rogue agents

Stop a looping, compromised or drifting agent from spreading harm: kill switches, budget and rate limits, circuit breakers, behavior monitoring, blast-radius limits, quarantine, rollback and incident response, all enforced outside the agent.

Design, Development, Maintenance · Platform engineers and SREs

Case study: Moffatt v. Air Canada, a chatbot that misstated the refund policy (2022-2024)

How an airline's website chatbot told a grieving customer they could claim a bereavement fare after travel, what the British Columbia Civil Resolution Tribunal decided in February 2024, and the grounding, testing and correction practices that keep an automated answer consistent with the policy it describes.

Design, Development, Maintenance · Developers

Insecure randomness lab: Generate unguessable API keys

An integration service issues 16-character API keys from Math.random, which is not built to be unpredictable. Take every character from the system’s secure generator, lengthen the key and map bytes without bias.