Exceptions and metrics
A security exception is a temporary, recorded decision to run with a known gap, approved by someone with authority over the consequence and ending on a set date. Program metrics show whether findings are fixed on time and whether the same weaknesses keep coming back.
Without expiry dates, accepted risks become permanent and drop out of view. Without defined metrics, a program can look healthy while hard findings age: a mean time to fix computed from closed findings leaves out every finding still open. Together they show leadership what risk is carried, by whom and until when, which no scanner or review reports.
- Who
- AppSec defines the metrics and the process; engineering owners request exceptions; business risk owners approve them
- When
- Whenever a finding will miss its deadline or a requirement cannot be met, and at each metrics review
- How often
- Expiring exceptions checked daily or weekly; metrics reviewed monthly or quarterly; the metrics themselves at least yearly
- You end up with
- An exception register with owners, tested controls and expiry dates, and a short dashboard of time to fix, backlog age and repeats
Do this
Set fix deadlines by risk tier
Write a deadline per tier, driven by exploitation and exposure rather than CVSS alone: days for a known-exploited flaw on an internet-facing service, the next scheduled upgrade for an internal one. Start the clock when the finding is confirmed and assigned. SAMM's Defect Management practice asks for one severity scheme with an SLA per class and regular reporting of breaches. See Risk-based prioritization.
Define what an exception record needs
Require the scenario, affected assets and versions, interim controls with test results, an expiry, triggers that reopen the decision and the risk owner. The requester never approves their own exception. SSDF's examples include handling requirement exception requests with periodic review of all approved exceptions (PO.1.2) and recording exception requests in the workflow (PO.4.1).
Credit only tested compensating controls
A compensating control earns credit only when a test of the deployed route shows it breaks the attack path, including alternate hostnames and direct backend access. Logging and alerts help detection and response; they do not prevent the attack, so they do not lower the risk being accepted.
Report exceptions as open risk
An approved exception is open risk with an owner, not a fix: the finding stays open with status "exception active". Flag exceptions that have expired or expire within 14 days, and when one is renewed, keep the original opening date so repeated deferral cannot reset its age.
Decide again at expiry
The risk owner chooses verified remediation, renewal on fresh evidence, or restricting the feature; nothing extends silently. A change in exploitation, data volume or access paths can force the decision before the date. SSDF's RV.2.2 examples treat risk acceptance as one possible response to a vulnerability and ask for records of approved exceptions to be kept up to date.
Define each metric before charting it
Write down the population, the start and stop events, the time window, and how duplicates, reopened findings and exceptions count. SAMM's Strategy and Metrics practice asks for metrics that are consistently measured, inexpensive to gather and expressed as a number or percentage, and for KPIs with goals and ranges that call for immediate intervention.
Report a few outcomes, then review them
A common set: SLA attainment for findings due in the period, open backlog by age, median and 90th-percentile time to fix that counts open findings, and repeats of the same CWE in the same service (Root cause analysis). Show counts beside every rate. SAMM's level-3 activity reviews KPIs at least yearly; drop any metric that changes no decision.
Done when
Metrics are reviewed on a set schedule, and every open exception has a risk owner and an expiry date.
Mark it in the gap checkCommon mistakes
- Counting an exception as a fix: the dashboard shows the risk as gone while it is still exploitable.
- Letting the requester or the security engineer approve the exception, so nobody with authority over the consequence has accepted it.
- Resetting a finding's age on renewal or re-import, which hides repeated deferral.
- Dividing on-time fixes by all closures instead of by all findings due. Overdue open findings drop out of both numbers, and the rate looks better than it is.
- Reporting mean time to remediate from closed findings only, so the slowest, still-open findings disappear from the figure.
- Ranking teams on raw counts such as findings closed. It invites closing duplicates, splitting tickets and switching off noisy scanners.
Which metrics, defined how?
Definitions as the guides use them. Agree them before the first chart, and annotate the chart when a definition or a scanner changes.
| Metric | Definition | Common trap |
|---|---|---|
| Time to remediate | Days from confirmed and assigned to a verified fix in production, per tier, as a median and 90th percentile that count open findings | Averaging closed findings only |
| SLA attainment | Findings due in the period and fixed by their due date, divided by all findings due in the period | Dividing by findings closed, which hides overdue open ones |
| Open backlog age | Open findings per tier in age buckets, plus overdue counts | Resetting age when a finding is reopened or re-imported |
| Repeats by CWE | Findings whose CWE was already fixed in the same service in the previous 365 days | Free-text categories that cannot be grouped |
| Escape rate | First-party findings first found in production, divided by all first-party findings confirmed in the period | Counting new third-party advisories as escapes |
| Exception load | Active, expiring, expired and renewed exceptions per tier, with their age | Counting an exception as a fix |
Tools and guides
- Workflow: Build an AppSec program
- Workflow: Respond to a vulnerability
- Approve, expire and measure security exceptions without hiding risk
Keep temporary acceptance separate from remediation and measure outcomes using definitions that survive changing visibility.
- Set remediation SLAs and measure AppSec outcomes
Set vulnerability SLAs by exploitation and exposure, route misses through expiring exceptions, and measure the program with cohort-based, censoring-aware metrics that resist gaming.
- Run an AppSec program: owners, finding states, deadlines and champions
Translate a security baseline into service ownership, change triggers, reusable patterns and evidence that work reaches deployment.
Standards
- NIST SSDF 1.1
- PO.1 Define Security Requirements for Software DevelopmentPO.4 Define and Use Criteria for Software Security ChecksRV.2 Assess, Prioritize, and Remediate Vulnerabilities
- OWASP SAMM 2
- Strategy and MetricsDefect Management
Sources
- OWASP SAMM: Defect Management, stream A (Defect Tracking)
- OWASP SAMM: Strategy and Metrics, stream B (Measure and Improve)
- NIST SP 800-218: Secure Software Development Framework (SSDF) Version 1.1. February 2022.
- NIST OSCAL content: SP 800-218 version 1.1 catalog (PO.1.2, PO.4.1 and RV.2.2 examples). oscal-content v1.5.0.