---
key: "product-incident-response"
title: "Respond to product security incidents: plan, tabletop and postmortem"
url: "https://softwaresecurity.com/domains/appsec/vulnerability-response/product-incident-response/"
api: "https://softwaresecurity.com/api/v1/articles/product-incident-response.json"
domain: "appsec"
topic: "Application security"
type: "Guide"
identifiers: []
tags: ["Incident response","AppSec programs","Compliance and assurance evidence"]
stages: ["respond","govern"]
roles: ["appsec-engineer","platform-sre","security-leader"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Respond to product security incidents: plan, tabletop and postmortem

An incident response plan for product security incidents (exploited vulnerabilities, compromised releases, leaked signing keys, malicious dependencies shipped to customers), aligned with NIST SP 800-61 Rev. 3, with reporting clocks, a two-scenario tabletop exercise template and a blameless postmortem template.

Canonical URL: https://softwaresecurity.com/domains/appsec/vulnerability-response/product-incident-response/

## In short

A product security incident is one where the product you ship or host is the problem: a vulnerability being exploited, a compromised build or release, a leaked signing key, or customer data exposed through a flaw. Prepare a short plan that names an incident commander, the PSIRT, engineering, communications and legal roles, defines severity, and lists the first-hour steps: record the moment of awareness, preserve evidence, contain, and check reporting clocks such as the EU CRA's 24-hour early warning. Rehearse it with tabletop exercises, and after every incident write a blameless postmortem with owned actions.

- Do record the moment of awareness first; CRA, GDPR and contract clocks start there, not when the fix ships.
- Do name one incident commander who decides and assigns, separate from the people debugging.
- Do preserve logs, artifacts and build records before changing systems, and keep a timestamped decision log.
- Do rehearse with tabletop exercises that use your real contacts and tools, and fix the gaps they find.
- Don't write postmortems that blame people: describe what people knew at the time and change the system.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| NIST incident response guidance | SP 800-61 Rev. 3, final 3 April 2025 | Supersedes Rev. 2 (2012) and organizes incident response around the six CSF 2.0 Functions. | NIST |
| EU CRA, actively exploited vulnerability or severe incident | Early warning 24 hours; notification 72 hours; final report later | Final report within 14 days of a corrective measure for a vulnerability, or one month after the notification for a severe incident. | Regulation (EU) 2024/2847, Article 14 |
| GDPR personal data breach | Notify the supervisory authority within 72 hours of becoming aware, where feasible | — | GDPR Article 33 |
| SEC registrants, material cybersecurity incident | Form 8-K Item 1.05 within four business days of determining materiality | — | SEC final rule, July 2023 |
| Postmortem | Within 10 business days of resolution (example) | Local target; actions get owners and due dates. | — |

## Verify it

1. Page the on-call incident commander and PSIRT lead at an unannounced time and measure how long it takes to open the incident channel.
2. Run one tabletop each half year and confirm every gap it found has an owner and a closed or scheduled action.
3. For your last incident, check that the timeline records the moment of awareness and that every notification was sent before its deadline.
4. Confirm the incident commander, legal and communications can reach the advisory, CVE and status-page tooling without help.
5. Read the last three postmortems and check that none names an individual as the cause and every action has an owner and a date.

## What you will learn

- Recognize product security incidents and hand off cleanly to and from the corporate CSIRT.
- Write a short incident response plan with roles, severity, a first-hour checklist and decision points.
- Identify the reporting clocks an incident can start and compute first-view deadlines.
- Run a tabletop exercise and write a blameless postmortem with owned actions.

Audience: PSIRT leads, engineering managers, SREs and AppSec engineers who respond to incidents involving their products

Before you start:

- Know who owns your products, releases and signing keys.
- Know which regulations and contracts apply to your company, or who in legal does.
- For the script: Python 3.11 or later.

Time: About 35 minutes: 8 to read, 5 on the code and 22 for the exercise, plus 90 for each tabletop scenario you run

## What counts as a product security incident?

A product security incident is one where your product, service or release process is the source of harm to customers, as opposed to an intrusion into your internal IT. The two often overlap: an attacker in your build system is both. Agree the boundary with the corporate CSIRT in writing, and define the hand-off in both directions.

| Incident | Typical first signal | Guide for the technical response |
| --- | --- | --- |
| Exploited vulnerability in a shipped product or hosted service | A researcher, a customer, CISA KEV or your own detection | /domains/runtime/detection/runtime-detection-response/ |
| Compromised build or release | A release with no matching commit or provenance | /domains/supply-chain/ci-cd/case-study-tj-actions/ |
| Leaked signing key, CI token or cloud credential | Secret scanning, a public paste, anomalous use | /domains/identity-crypto/secrets/leaked-secret-response/ |
| Malicious dependency shipped to customers | A registry or vendor advisory, unexpected network calls | /domains/supply-chain/dependencies/package-attack-response/ |
| Customer data exposed through a product flaw | A report of cross-tenant access, logs, a customer complaint | /domains/appsec/threat-modeling/multitenant-design-review/ |

## How does NIST SP 800-61 Rev. 3 frame incident response?

NIST SP 800-61 Rev. 3, final in April 2025, replaces the 2012 four-phase life cycle with a model built on the six CSF 2.0 Functions. Govern, Identify and Protect are the preparation that makes response possible; Detect, Respond and Recover are the incident response itself; and Identify's Improvement category takes lessons from every activity and feeds them back at any time, not only after recovery. NIST notes that incidents now happen often and can take weeks or months to recover from, so response has to be part of risk management rather than a separate team's occasional job.

For a product team the practical reading is: preparation lives in your secure development and PSIRT work, response is what this plan covers, and improvement is the postmortem plus the changes you make before the next incident.

| Earlier SP 800-61 phase | CSF 2.0 Functions in Rev. 3 | In a product organization |
| --- | --- | --- |
| Preparation | Govern, Identify, Protect | This plan, the PSIRT, logging in the product, tabletop exercises |
| Detection and analysis | Detect, plus Identify (Improvement) | Reports, detections, triage, severity |
| Containment, eradication and recovery | Respond, Recover, plus Identify (Improvement) | Mitigation, fix, release, customer guidance |
| Post-incident activity | Identify (Improvement) | Postmortem and follow-up actions |

## Who does what during an incident?

Keep the roles few and explicit. The incident commander decides and assigns but does not debug; the scribe keeps the timeline that later proves when you knew and what you did. The PSIRT lead owns the vulnerability facts, the advisory and the CVE, as described in /domains/appsec/vulnerability-response/psirt-operating-model/.

| Role | Owns | Hands off to |
| --- | --- | --- |
| Incident commander | Severity, decisions, priorities, the update rhythm | Executive sponsor for decisions beyond their authority |
| PSIRT lead | Vulnerability facts, finder contact, advisory and CVE, vendor coordination | Communications for customer wording |
| Engineering lead | Containment, fix, rollback, technical forensics | Incident commander for trade-offs |
| Communications | Customer notices, status page, press; one voice | Legal for approval |
| Legal and privacy | Reporting obligations and deadlines, privilege, law enforcement | Incident commander for timing |
| Support lead | Customer questions, scripts, tagging related tickets | Communications for new facts |
| Scribe | Timestamped log of facts, decisions and actions | Postmortem authors |

## What goes in the incident response plan?

Keep the plan short enough to use at 2 a.m.: scope, severity levels, roles, the first-hour checklist, the decision points and when the incident closes. Store it in version control next to the runbooks it points to, and review it after every incident and tabletop. The skeleton below is a starting point; replace the severity definitions and times with your own.

```yaml
# Product security incident response plan (skeleton). Keep it in version control; review it after every incident and tabletop.
scope:
  covers: [exploited vulnerability in a shipped product or hosted service, compromised build or release,
           leaked signing key or CI credential, malicious dependency shipped to customers,
           customer data exposed through a product flaw]
  hands_off_to: corporate CSIRT for incidents in internal IT only
severity:
  SEV1: Active exploitation, or customer data or release integrity compromised. Page now; executive informed within 1 hour.
  SEV2: Credible exploit path to SEV1 impact, no evidence of exploitation yet. Respond within 4 hours.
  SEV3: Contained or low impact. Next business day.
roles:
  incident_commander: decides, assigns, keeps the timeline; does not debug
  psirt_lead: vulnerability facts, advisory, CVE, coordination with finders and vendors
  engineering_lead: containment, fix, rollback, forensics with the security team
  communications: customer notices, status page, press; one voice
  legal_privacy: regulatory clocks (CRA, GDPR, SEC, contracts), privilege, law enforcement
  support_lead: customer questions, scripts, ticket tagging
  scribe: timestamps every decision and action in the incident channel
first_hour:
  - Open the incident channel and the timeline document; name the commander and scribe.
  - Record the moment of awareness (it starts regulatory clocks).
  - Preserve evidence before changing systems (logs, images, build records, tokens' audit trails).
  - Decide containment (feature flag, WAF rule, revoke credential, pull release) and who does it.
  - Ask legal whether CRA, GDPR, SEC or contract notifications may apply; run deadlines.py.
decision_points:
  - Do customers need to act? If yes: advisory, notification and CVE.
  - Is a release or signing key compromised? If yes: revoke, rotate, re-sign, publish verification steps.
  - Is personal data involved? If yes: privacy assessment and GDPR clock.
  - Is the vulnerability actively exploited and the product in CRA scope? If yes: early warning within 24 hours.
closure:
  - Fix deployed or released for every supported version; advisory published; CVE record published.
  - Blameless postmortem held within 10 business days; actions have owners and dates.
```

## What should happen in the first hour?



- Record the moment of awareness and who learned what; reporting clocks start there.
- Name the incident commander and scribe, open the channel and the timeline.
- Preserve evidence before changing anything: logs, container images, build and release records, audit trails of the credentials involved.
- Contain with the smallest effective action: a feature flag, a WAF rule, revoking a token, yanking a release.
- Decide whether customers must act; if so, start the advisory and the customer notice in parallel with the fix.
- Ask legal to confirm which reporting obligations apply and when; run the deadline script below for a first view.

## Which reporting clocks can start?

Several obligations can apply to one incident, each with its own trigger and deadline. Under Article 14 of the EU Cyber Resilience Act, from 11 September 2026 a manufacturer must send an early warning within 24 hours of becoming aware of an actively exploited vulnerability or a severe incident affecting a product in scope, a notification within 72 hours, and a final report later; see /frameworks/cra-reporting-runbook/. A personal data breach starts the GDPR Article 33 clock of 72 hours where feasible. A US public company that determines an incident is material files Form 8-K Item 1.05 within four business days of that determination. Contracts often add their own notice periods.

The script gives a first view of the deadlines from the awareness time and an optional materiality determination. For an auth bypass whose exploitation was discovered at 18:25 UTC on Friday 16 October 2026, with materiality determined on Monday at 10:00, it shows the CRA early warning due on Saturday at 18:25, the 72-hour notifications on Monday at 18:25 and the Form 8-K on Friday 23 October. It counts weekends but not public holidays, and it is not legal advice: legal decides which obligations apply.

```python
"""Illustrative reporting deadlines from the moment of awareness. Not legal advice.

Usage: python deadlines.py 2026-10-16T17:40:00+00:00 [--materiality 2026-10-19T10:00:00+00:00]
Business days skip Saturdays and Sundays only; add public holidays for real use.
"""
import argparse
from datetime import datetime, timedelta


def add_business_days(start, days):
    current = start
    while days:
        current += timedelta(days=1)
        if current.weekday() < 5:  # Monday to Friday
            days -= 1
    return current


def deadlines(aware, materiality=None):
    rows = [
        ("EU CRA early warning (actively exploited vulnerability or severe incident)", aware + timedelta(hours=24)),
        ("EU CRA notification", aware + timedelta(hours=72)),
        ("GDPR Article 33 notification to the supervisory authority (personal data breach)", aware + timedelta(hours=72)),
    ]
    if materiality:
        rows.append(("SEC Form 8-K Item 1.05 (four business days after the materiality determination)",
                     add_business_days(materiality, 4)))
    return rows


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("aware", type=datetime.fromisoformat)
    parser.add_argument("--materiality", type=datetime.fromisoformat)
    args = parser.parse_args()
    for label, due in deadlines(args.aware, args.materiality):
        print(f"{due:%a %Y-%m-%d %H:%M %Z}  {label}")
```

## How should you communicate with customers during an incident?



- Say what you know, what you do not know yet, what customers should do now and when the next update will come.
- Never claim "no data was accessed" unless logs covering the whole window prove it; say what the logs cover instead.
- Use one channel of record (status page or advisory) and point every other message to it.
- Give concrete actions: versions to install, credentials to rotate, indicators to search for.
- Publish the advisory and CVE record as described in /domains/appsec/vulnerability-response/security-advisories-csaf-vex/ once the fix or mitigation exists.

## How do you run a tabletop exercise?

A tabletop is a facilitated discussion of a realistic scenario, delivered as timed injects, that tests decisions rather than technical skill. CISA publishes Tabletop Exercise Packages (CTEPs) with scenarios, facilitator guides and evaluation material that you can adapt. Use your real contact lists, chat tools and runbooks, keep the discussion blameless, record every decision with its time, and end with a list of gaps, each with an owner.

The template below has two 90-minute scenarios built for product teams. The first starts with a researcher's report late on a Friday that turns into exploitation; the second starts with a leaked publish token and a malicious release, like the incidents in /domains/supply-chain/ci-cd/case-study-tj-actions/ and /domains/supply-chain/dependencies/case-study-xz-utils/.

```yaml
# Tabletop exercise template with two scenarios. 90 minutes each; a facilitator reads injects in order.
objectives:
  - Test the first hour of the product IR plan, including who decides and how fast.
  - Practise the regulatory-clock check and the customer-communication decision.
  - Find gaps in access, contacts and runbooks, and turn them into actions.
participants: [incident commander, PSIRT lead, engineering lead, communications, legal and privacy, support lead, scribe]
ground_rules: [no-fault discussion, use real tools and contact lists, the facilitator may add injects, decisions are recorded with times]
scenarios:
  - name: Auth bypass in the SaaS API, reported then exploited
    injects:
      - {t: "0:00 (Fri 17:40)", event: "A researcher reports via the VDP that a crafted JWT with alg none is accepted by /api/v2.", expect: "Acknowledge, triage owner, reproduce in staging."}
      - {t: "0:20", event: "Reproduced: any tenant's admin API is reachable with a forged token.", expect: "SEV2; containment options; who can deploy on a Friday evening?"}
      - {t: "0:45", event: "Logs show forged tokens from two IP addresses since Thursday.", expect: "SEV1; awareness of exploitation recorded; CRA and GDPR check; evidence preserved."}
      - {t: "1:10", event: "A customer's CISO emails asking whether they are affected.", expect: "Holding statement from communications; no speculation."}
      - {t: "1:30", event: "Fix ready; the researcher asks to publish on Monday.", expect: "Agree disclosure date, CVE, advisory and customer notice; plan the postmortem."}
    evaluate: [time to SEV1 decision, evidence preserved before changes, clocks identified, one voice to customers]
  - name: Leaked CI token and a malicious update
    injects:
      - {t: "0:00", event: "Secret scanning alerts on a CI token with package-publish rights, posted in a public gist 6 hours ago.", expect: "Revoke the token now; who owns it?"}
      - {t: "0:15", event: "Registry shows version 3.4.3 of the SDK published 2 hours ago by that token; no matching tag in Git.", expect: "SEV1; yank or deprecate 3.4.3; preserve the artifact."}
      - {t: "0:40", event: "Analysis: 3.4.3 sends environment variables to an external host on install.", expect: "Advisory and malicious-package report; customers told to rotate secrets; downloads counted."}
      - {t: "1:00", event: "A partner says their production build pulled 3.4.3.", expect: "Coordinate; give indicators of compromise and safe version."}
      - {t: "1:20", event: "Journalist asks for comment.", expect: "Communications answers from the approved facts only."}
    evaluate: [time to revoke, time to yank, quality of customer instructions, provenance gaps that allowed the publish]
outputs: [decision log with times, list of gaps with owners, updates to the plan and runbooks]
```

## What goes in a blameless postmortem?

Google's SRE book defines a blameless postmortem as one that identifies contributing causes "without indicting any individual or team for bad or inappropriate behavior", assuming people acted with good intentions on the information they had. Its common triggers include user-visible impact beyond a threshold, data loss of any kind, on-call intervention such as a rollback, resolution time above a threshold and monitoring failures. For security, add any confirmed exploitation, any credential or signing-key exposure, and any customer notification.

A senior reviewer should check that incident data was kept, the impact assessment is complete, the causes go deep enough, the actions are appropriate and prioritized, and the outcome was shared. As the SRE book puts it, "an unreviewed postmortem might as well never have existed." Feed security causes back into requirements, tests and training, for example through /domains/appsec/threat-modeling/security-requirements-definition-of-done/.

```markdown
# Postmortem: <incident title> (<incident id>)

Status: draft | in review | final     Severity: SEV1 | SEV2 | SEV3
Authors: <team>     Incident commander: <name>     Date of incident: <YYYY-MM-DD>

## Summary
Two or three sentences: what happened, impact on customers, how it ended.

## Impact
- Customers and products affected, with versions and time window.
- Data exposed or altered, and how that was determined.
- Notifications and filings made (advisory, CVE, CRA, GDPR, contracts), with times.

## Timeline (UTC)
| Time | Event or decision | Source |
| --- | --- | --- |
| | Awareness: when and how we learned | |
| | Containment | |
| | Fix released or deployed | |
| | Customers notified | |

## Contributing factors
Conditions in the system, process and tools that made the incident possible or worse.
Describe what people knew at the time; do not name or blame individuals.

## What went well

## What was hard or lucky

## Action items
| Action | Type (prevent / detect / respond) | Owner | Due | Ticket |
| --- | --- | --- | --- | --- |

## Follow-up
Review date for open actions. Link the updated plan, runbooks and tests.
```

## Where does product incident response go wrong?



- No one writes down when the team became aware, so nobody can show the reporting deadlines were met.
- Engineers rebuild or patch the affected system before collecting evidence.
- The incident commander is also the main debugger and stops making decisions.
- Customers hear about the incident from a researcher or the press first.
- The plan exists but has never been exercised, and the contact list is a year out of date.
- Postmortem actions have no owners or dates, so the same incident happens again.

## Guided practice

Run the first scenario of the tabletop on paper for a fictional SaaS company, Ledgerly. Decide severity, the clocks and the first customer message, then draft the postmortem summary.

### Supplied material

```text
Ledgerly (fictional): SaaS invoicing; EU and US customers; US-listed company; product in scope of the EU CRA.
Fri 2026-10-16 17:40 UTC  VDP report: /api/v2 accepts JWTs with alg "none"; researcher includes a forged token.
Fri 18:00  Reproduced in staging: any tenant's admin API reachable.
Fri 18:25  Logs: forged tokens used from two IP addresses since Thu 2026-10-15 03:12 UTC; requests read invoices of 14 tenants, including names and email addresses.
Fri 19:30  Fix rejecting alg "none" deployed; forged tokens now return 401.
Mon 2026-10-19 10:00  Executives determine the incident is material.
Draft customer message from sales: "We had a minor issue; no data was accessed."
```

### 1. Set severity and awareness

When does the incident become SEV1, and what is the moment of awareness for exploitation?

Worked answer: SEV2 at 18:00 when reproduced; SEV1 at 18:25 when logs show exploitation and data access. Awareness of exploitation is Fri 18:25 UTC; record it in the timeline.

### 2. List the clocks

Which reporting deadlines apply, assuming legal confirms each obligation?

Worked answer: CRA early warning by Sat 2026-10-17 18:25 UTC and notification by Mon 2026-10-19 18:25 UTC; GDPR notification within 72 hours where feasible, also Mon 18:25, because personal data was read; Form 8-K Item 1.05 by Fri 2026-10-23, four business days after the Monday materiality determination.

### 3. Fix the customer message

What is wrong with the draft, and what should the first message say?

Worked answer: It is false: logs show invoices of 14 tenants were read. Say what happened, the window (from Thu 03:12 UTC), that a fix was deployed Fri 19:30 UTC, that affected tenants will be contacted directly, what customers should do, and when the next update comes.

### 4. Plan the advisory

Is a CVE and advisory needed for a SaaS-only flaw?

Worked answer: Yes, if customers need to act or for transparency; CNA rules allow CVEs for hosted services and provide an `exclusively-hosted-service` tag. Publish the advisory with the fix and affected window.

### 5. Draft the postmortem summary

Write the summary and two contributing factors without blaming individuals.

Worked answer: Summary: forged JWTs with `alg` `"none"` were accepted by `/api/v2` from Thu 03:12 to Fri 19:30 UTC; invoices of 14 tenants were read; fixed by rejecting unsigned tokens. Contributing factors: the JWT library was configured without an algorithm allowlist; no test asserted that unsigned tokens fail (see /domains/identity-crypto/authentication/jwt-attacks/).

Deliverable: A one-page incident timeline with severity changes, the list of deadlines with times, a corrected first customer message and a postmortem summary with two contributing factors.

Scope: Ledgerly and its incident are fictional. Whether CRA, GDPR or SEC obligations apply depends on facts that legal must assess; the deadlines shown assume each applies.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [NIST SP 800-61 Rev. 3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management](https://csrc.nist.gov/pubs/sp/800/61/r3/final). April 2025.
- [CISA: Tabletop Exercise Packages (CTEPs)](https://www.cisa.gov/resources-tools/services/cisa-tabletop-exercise-packages)
- [Google SRE book: Postmortem Culture: Learning from Failure](https://sre.google/sre-book/postmortem-culture/)
- [Google SRE workbook: Postmortem Culture](https://sre.google/workbook/postmortem-culture/)
- [Regulation (EU) 2024/2847 (Cyber Resilience Act)](https://eur-lex.europa.eu/eli/reg/2024/2847/oj). Official Journal, 20 November 2024.
- [Regulation (EU) 2016/679 (GDPR), Article 33](https://eur-lex.europa.eu/eli/reg/2016/679/oj). Official Journal, 4 May 2016.
- [SEC: final rules on cybersecurity incident disclosure (press release 2023-139)](https://www.sec.gov/newsroom/press-releases/2023-139). 26 July 2023.
- [FIRST: PSIRT Services Framework 1.1 (Service 4.3 Incident Handling)](https://www.first.org/standards/frameworks/psirts/psirt_services_framework_v1.1)