--- key: "appsec-metrics-and-slas" title: "Set remediation SLAs and measure AppSec outcomes" url: "https://softwaresecurity.com/frameworks/appsec-metrics-and-slas/" api: "https://softwaresecurity.com/api/v1/articles/appsec-metrics-and-slas.json" domain: "governance" topic: "Governance \u0026 frameworks" type: "Guide" identifiers: ["V15"] tags: ["Vulnerability management","AppSec programs","Secure development lifecycle"] stages: ["respond","govern"] roles: ["appsec-engineer","platform-sre","security-leader"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Set remediation SLAs and measure AppSec outcomes Set vulnerability SLAs by exploitation and exposure, route misses through expiring exceptions, and measure the program with cohort-based, censoring-aware metrics that resist gaming. Canonical URL: https://softwaresecurity.com/frameworks/appsec-metrics-and-slas/ ## In short Measure outcomes, not activity. Report time to remediate per tier as a median and 90th percentile that count still-open findings, the age of the open backlog, SLA attainment for findings due in the period, escape rate, coverage of critical services and repeats by CWE. Set SLA tiers by exploitation and exposure rather than CVSS alone, start the clock when the finding is confirmed and assigned, and route misses to exceptions that expire. - Do define each metric's population, start event, stop event and time window before you chart it. - Do report medians and 90th percentiles that include open findings; a mean of closed tickets hides the slow ones. - Do compute SLA attainment for the cohort due in the period; late and still-open findings both count as missed. - Do tie SLA tiers to exploitation and exposure, give triage its own target, and keep the first detection date when a finding reopens. - Don't count an exception as a fix: it is open risk with an owner and an expiry. - Don't publish a rate without its counts, or a metric that changes no decision. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Triage target | Confirm, reject or route within 2 business days; same day if KEV-listed | Local starting point; measure detection-to-assignment time. | — | | Exploited, total impact, publicly exposed or automatable | Mitigate the same day; fix within 3 calendar days; check for compromise | Fix deadline and compromise check match BOD 26-04; the same-day mitigation target is local. | CISA BOD 26-04 | | Other urgent findings | 3 calendar days | — | CISA BOD 26-04 | | High | 14 calendar days | — | CISA BOD 26-04 | | Planned | 60 calendar days | — | CISA BOD 26-04 | | Routine (not exposed, not automatable, not exploited) | Next scheduled upgrade, at most 180 days | BOD 26-04 has no cap; the 180-day limit is local. | CISA BOD 26-04 | | Time-to-remediate statistic | Kaplan-Meier median and 90th percentile per tier, open findings censored | — | NIST/SEMATECH e-Handbook 8.2.1.5 | | Escape-rate scope | First-party code and configuration only | New third-party advisories are disclosures, not escapes. | — | | Repeat window | Same CWE in the same service within 365 days after a fix | — | — | ## Verify it 1. Recompute last quarter's SLA attainment from raw finding records with the cohort definition and compare it with the dashboard. 2. Sample ten closed findings and confirm each has a deployed fix or verified absence, not just a status change. 3. Reopen or re-import a test finding and confirm its first detection date and age survive. 4. List exceptions and confirm each has an owner with authority, an expiry and a tested compensating control, and that expired ones show as overdue. 5. Compare the tier-1 service inventory with each tool's project list and name the services no tool covers. 6. Run the metric code on the exercise fixture and check the results against the expected answers. ## What you will learn - Compute cohort SLA attainment and backlog age without hiding overdue work. - Estimate time to remediate with Kaplan-Meier so open findings count. - Write an SLA policy with triage, tiers, clock rules and expiring exceptions. - Separate first-party escapes and CWE repeats from third-party disclosures. Audience: AppSec leads, engineering managers and platform owners who set vulnerability SLAs or report program metrics Before you start: - Access to finding records with detection, due and closure dates. - A tiering policy, such as a decision table built on CISA BOD 26-04. - For the code: Python 3.11 or later and SQLite. Time: About 36 minutes: 12 to read and 24 for the exercise ## Which AppSec metrics are worth reporting? A metric earns its place when a change in it changes what someone does: fund a fix, unblock a team, adjust a control or accept a risk. Counts of scans run, findings detected or training hours describe effort. They are useful context, but they cannot show whether exposure went down. OWASP SAMM's Strategy and Metrics practice sorts metrics into effort, result and environment, and recommends combining them: outstanding high-severity defects per application says more than either count alone. It also asks for metrics that are measured consistently, cheap to gather, and expressed as a number or percentage with a unit. NIST SSDF 1.1 asks for the same discipline. Task PO.4.1 calls for defined criteria for software security checks; its examples include KPIs, key risk indicators and severity scores, and recording approvals, rejections and exception requests in the workflow. RV.2 covers risk-based remediation and RV.3 root-cause analysis over time. BSIMM16, built from observations of 111 organizations, records how firms do this work under its Strategy & Metrics and Configuration Management & Vulnerability Management practices. None of these frameworks sets target numbers. The definitions and targets below are local choices to adapt. ## Define each metric before you chart it Most arguments about metrics come from undefined populations and timestamps. For each metric, write down the population, the start and stop events, the time window, and how duplicates, reopened findings and exceptions are handled. Use the same definitions for every team, and annotate the chart whenever a definition or a scanner changes. | Metric | Definition | Why it matters | Common trap | | --- | --- | --- | --- | | Time to remediate | Days from clock start to a verified fix in production, per tier; Kaplan-Meier median and 90th percentile | Shows whether SLAs are realistic and where the slow tail is | Averaging closed findings only, which drops the slowest | | SLA attainment | Findings due in the period and fixed by their due date, divided by all findings due in the period | The promise made to the business | Dividing by findings closed, which hides overdue open ones | | Open backlog age | Open findings per tier in age buckets (0-30, 31-60, 61-90, 91-180, over 180 days), plus overdue counts | Shows debt that closure rates hide | Resetting age when a finding is reopened or re-imported | | Window of exposure | Days a vulnerable version was deployed in production | Tracks exposure rather than ticket state | Closing on merge instead of deployment | | Escape rate | First-party findings first found in production, divided by all first-party findings confirmed in the period | Tests whether design review and pre-release testing work | Counting new third-party advisories as escapes | | Coverage | Tier-1 services with each required control, divided by all tier-1 services in the inventory | Finds services that no tool or review touches | Using scanned repositories as the denominator | | Repeats by CWE | Findings whose CWE was already fixed in the same service in the previous 365 days | Points to missing shared fixes and training | Free-text categories that cannot be grouped | | Exception load | Active, expiring, expired and renewed exceptions per tier, with their age | Shows risk carried rather than removed | Counting an exception as a fix | ## How do you report time to remediate without hiding open findings? The usual mean time to remediate is computed from closed findings only. That drops the findings you most need to see: the ones still open. A team that fixes easy items quickly and leaves hard ones open looks fast. Treat open findings as censored observations, the way reliability engineers treat units still running when a test ends, and estimate the time-to-fix distribution with the Kaplan-Meier product-limit method. In the example, ten findings in the 14-day tier opened last quarter and four are still open. The median of the six closed findings is 10 days. The Kaplan-Meier median, which keeps open findings in the risk set until their current age, is 30 days. The 90th percentile cannot be estimated yet because too many findings are still open, so report it as not reached rather than inventing a number. The function agrees with the lifelines library except when the curve lands exactly on the quantile, where tools apply different tie rules. ```python def km_quantile(durations, closed, q=0.5): """Kaplan-Meier estimate of the q-quantile of time to remediate. durations: days from clock start to closure, or to today if still open. closed: True if remediated, False if still open (censored). Returns None when too many findings are still open to reach q. """ pairs = sorted(zip(durations, closed)) at_risk, survival, i = len(pairs), 1.0, 0 while i < len(pairs): t, events, ties = pairs[i][0], 0, 0 while i < len(pairs) and pairs[i][0] == t: events, ties, i = events + pairs[i][1], ties + 1, i + 1 if events: survival *= 1 - events / at_risk if survival <= 1 - q + 1e-9: return t at_risk -= ties return None if __name__ == "__main__": from statistics import median days = [3, 5, 8, 12, 20, 30, 41, 45, 60, 75] # T3 (14-day) findings, last quarter closed = [True, True, True, True, False, True, False, False, True, False] print("median of closed only:", median(d for d, c in zip(days, closed) if c)) print("Kaplan-Meier median:", km_quantile(days, closed)) print("Kaplan-Meier p90:", km_quantile(days, closed, 0.9)) ``` ## How do you compute SLA attainment and backlog age? SLA attainment answers one question: of the findings due in the period, how many were fixed on time? Build the cohort from due dates, and count late closures and still-open findings as missed. List findings missed under an active exception separately, but never move them into the met column. In the exercise fixture, seven findings were due in September 2026 and four were fixed by their due dates, so attainment is 4 of 7, or 57%. Dividing on-time closures by all closures gives 80%, because the two overdue open findings drop out of both numbers. Report the open backlog beside it: the age buckets surface a 122-day-old finding that no September figure mentions. ```python import csv from datetime import date def load(path): rows = list(csv.DictReader(open(path, newline=""))) for r in rows: for k in ("confirmed", "due", "closed", "exception_until"): r[k] = date.fromisoformat(r[k]) if r[k] else None return rows def sla_attainment(findings, start, end): """Cohort: findings whose due date fell in [start, end]. Closed late or still open both count as missed.""" cohort = [f for f in findings if start <= f["due"] <= end] met = [f for f in cohort if f["closed"] and f["closed"] <= f["due"]] missed = [f for f in cohort if f not in met] excepted = [f for f in missed if f["exception_until"]] return {"met": len(met), "cohort": len(cohort), "missed": [f["id"] for f in missed], "missed_with_exception": [f["id"] for f in excepted]} def open_age(findings, as_of, edges=(30, 60, 90, 180)): """Open findings by days since the clock started, plus overdue ones.""" buckets = {f"<={e}": 0 for e in edges} | {f">{edges[-1]}": 0} overdue = [] for f in findings: if f["closed"] is None: age = (as_of - f["confirmed"]).days buckets[next((f"<={e}" for e in edges if age <= e), f">{edges[-1]}")] += 1 if f["due"] < as_of: overdue.append(f["id"]) return buckets, overdue if __name__ == "__main__": findings = load("findings.csv") print(sla_attainment(findings, date(2026, 9, 1), date(2026, 9, 30))) print(open_age(findings, as_of=date(2026, 10, 1))) ``` ## What SLAs can engineers actually meet? Engineers accept SLAs that follow real risk, start when they could reasonably know, and fit their delivery capacity. Use the tiers from your prioritization decision table (/frameworks/vulnerability-prioritization/) rather than CVSS bands alone, so a KEV-listed, internet-facing finding gets days and an internal, non-automatable one waits for the next upgrade. Count calendar days for remediation; attackers do not wait for Monday. Give triage its own target, start the remediation clock when the finding is confirmed and assigned to the owning team, and measure detection-to-assignment time so routing delays stay visible. For exploited vulnerabilities, start at the earlier of the KEV listing and detection on the asset, as BOD 26-04 does. Check feasibility before you publish. Compare each proposed deadline with the last two quarters' Kaplan-Meier 90th percentile for that tier. If the gap is large, invest in the fix path, such as automated dependency updates, base-image rebuilds or more frequent releases, or phase the target in. A deadline most teams miss teaches them to ignore it. Separate mitigation from remediation: when no vendor fix exists, the mitigation deadline still applies and remediation continues under an exception. Decide whether a reopened finding restarts its clock, and keep the original detection date for age and exposure metrics either way. Tools differ here: DefectDojo, for example, counts from the finding's creation date, does not restart on reimport, and offers a setting to restart on reactivation. SAMM's Defect Management practice describes the progression. At maturity level 2, one rating method applies across the organization, each severity class has an SLA, breaches are reported centrally, and a process handles defects that cannot be fixed in time. At level 3, SLA breaches raise automated alerts and move into risk management. ASVS 5.0.0 requirements 15.1.1 and 15.2.1, both level 1, ask an application to document risk-based remediation time frames for third-party components and to contain no component that has breached them. ## Use an SLA policy template Adapt the values and keep the structure. The fix deadlines follow BOD 26-04; the triage target, the same-day mitigation target, the 180-day cap and the exception limits are local choices. Write down the rules a table cannot hold: - Clock: starts when the finding is confirmed and assigned; for exploited vulnerabilities, at the earlier of the KEV listing and detection on the asset. - Change: when a signal changes, re-tier the finding and keep the earlier of the old due date and the change date plus the new period. - Closure: a verified fix deployed to production, or verified absence of the component. A merged pull request is not closure. - Grouping: plan capacity by fix, so a base-image rebuild that clears 300 findings is one unit of work. - Reopening: keep the first detection date and record a reopen count. | Tier | Applies when | Mitigate by | Fixed in production by | Exception approver | Longest exception | | --- | --- | --- | --- | --- | --- | | Triage | Every new finding | - | Confirmed, rejected or routed within 2 business days; same day if KEV-listed | - | - | | T1 Emergency | Exploited, total impact, and publicly exposed or automatable | Same day | 3 days, plus a compromise check | Executive risk owner | 7 days | | T2 Urgent | Exploited, public, automatable, partial impact; or public, automatable, total impact | - | 3 days | Business owner of the service | 14 days | | T3 High | Other exploited findings; or public and either automatable or total impact | - | 14 days | Business owner of the service | 30 days | | T4 Planned | Public, not automatable, partial impact; or internal and automatable | - | 60 days | Engineering manager | 90 days | | T5 Routine | Internal, not automatable, not exploited | - | Next scheduled upgrade, at most 180 days | Engineering manager | 180 days | | Not affected | Component verified absent from the deployed artifact | - | No clock; re-check when the build or the advisory changes | AppSec reviewer | - | ## Route misses through exceptions that expire An SLA with no exception path gets bypassed, and an exception path with no limits becomes the real SLA. Require an owner with authority over the business consequence, the reason the fix cannot ship in time, a tested compensating control, a trigger that reopens the decision, and an expiry no later than the tier's limit. At expiry the owner chooses remediation, renewal with new evidence, or restricting the feature. A renewal keeps the original age. In metrics, an exception is open risk, not a fix. Show active, expiring within 30 days, expired and renewed exceptions by tier, and keep excepted findings in the missed column of SLA attainment. Recording exception requests in the workflow, one of SSDF PO.4.1's examples, gives you the data. /frameworks/risk-exceptions-and-metrics/ covers the decision record and the cohort arithmetic in more depth. ## How do you measure escapes and repeat findings? Escape rate asks how many first-party vulnerabilities reached production before anyone found them: through external reports or bug bounty, production penetration tests, or incidents. Limit it to your own code and configuration. A new advisory in a dependency is a new disclosure, not a failure of your pre-release testing, and including it makes the metric track the advisory feed. Repeats by CWE show where fixes stay local. Tag every first-party finding with a CWE ID at triage, then count findings whose CWE was already fixed in the same service in the previous year. SSDF tasks RV.3.2 and RV.3.3 ask for this pattern analysis and for removing whole classes of vulnerability rather than single instances. In the fixture, three of the six first-party findings confirmed in Q3 2026 were found in production, an escape rate of 50%, and the checkout service repeated CWE-639, Authorization Bypass Through User-Controlled Key, 25 days after fixing it. A shared, tested authorization check is the class fix. Repeat findings by CWE and team are also the outcome measure for developer training; see /frameworks/developer-security-training/. ```sql -- SQLite; dates are ISO-8601 text and closed is NULL while a finding is open. -- Escape rate: first-party findings confirmed in Q3 2026 that were -- first found in production. Third-party advisories are new disclosures, -- not escapes, so they are excluded. SELECT COUNT(*) AS confirmed, SUM(found_in = 'production') AS escaped, ROUND(100.0 * SUM(found_in = 'production') / COUNT(*), 1) AS escape_pct FROM findings WHERE origin = 'first-party' AND confirmed BETWEEN '2026-07-01' AND '2026-09-30'; -- Recurrence: the same CWE confirmed again in the same service within -- 365 days after an earlier finding of that CWE was fixed. SELECT f.cwe, COUNT(*) AS repeats_after_fix FROM findings f WHERE f.origin = 'first-party' AND f.confirmed BETWEEN '2026-07-01' AND '2026-09-30' AND EXISTS ( SELECT 1 FROM findings p WHERE p.service = f.service AND p.cwe = f.cwe AND p.id <> f.id AND p.closed IS NOT NULL AND p.closed < f.confirmed AND p.closed >= date(f.confirmed, '-365 days')) GROUP BY f.cwe ORDER BY repeats_after_fix DESC; ``` ## Measure coverage against the service inventory A coverage figure is only as good as its denominator. Start from an inventory of production services with owners and tiers, not from the repositories a scanner happens to know. For each tier-1 service, track whether each required control is in place and current: dependency scanning on the default branch, a code-scanning or review gate, an SBOM for each release, a threat model reviewed in the last 12 months, and a penetration test or DAST run against the current major version. Report the gaps by name. A 92% figure does not say which services make up the missing 8%. ## How do you build a dashboard that resists gaming? A dashboard turns definitions into behavior, so design each panel for the decision it should drive. Show raw counts beside every rate, prefer cohorts and distributions to averages, and annotate scope changes, such as a new scanner or newly onboarded services, that raise counts without any change in risk. SAMM's guidance on KPIs asks for documented data sources, short- and long-term goals, and ranges that call for immediate intervention, so give each panel an owner and a threshold. Leave out numbers that reward the wrong work: total findings found, scans run, percentage closed without a cohort, mean time to remediate of closed items, and sums of CVSS scores. Ranking teams on these invites closing duplicates, splitting tickets and switching off noisy scanners. Compare each team with its own history and its own tier mix instead. A small set of panels covers most decisions: - SLA attainment by tier for the period, shown as met out of due. - Open backlog by tier and age bucket, with the oldest findings listed by name. - Kaplan-Meier median and 90th-percentile time to remediate by tier, per quarter. - Exceptions: active, expiring within 30 days, expired and renewed. - Coverage of tier-1 services, with the gaps named. - Escape rate and the most repeated CWEs. ## Check the data before you trust the chart Recompute one headline number from raw records each quarter and compare it with the dashboard. Sample closed findings and confirm that each has a deployed fix or verified absence, not just a status change. Check that reopened and re-imported findings kept their first detection date, that every exception has an owner and an expiry, and that deduplication has not merged findings across services. At SAMM maturity level 3, the Metrics and Feedback stream asks for the same: automated data-quality checks, and a yearly review that drops metrics whose value does not justify the effort of collecting them. ## Guided practice You own the monthly AppSec report for four services. Compute September 2026 SLA attainment and the Q3 metrics from the finding records, then fix a proposed dashboard. Services and findings are fictional. ### Supplied material ```text As of 2026-10-01. Tier names are due-date periods; F6 has an approved exception until 2026-10-26. id,service,tier,origin,cwe,found_in,confirmed,due,closed,exception_until F1,checkout,14d,first-party,CWE-639,pre-release,2026-08-20,2026-09-03,2026-09-02, F2,checkout,60d,third-party,,pre-release,2026-07-10,2026-09-08,2026-09-15, F3,search,3d,third-party,,production,2026-09-10,2026-09-13,2026-09-12, F4,search,14d,first-party,CWE-79,production,2026-09-05,2026-09-19,, F5,billing,14d,first-party,CWE-639,production,2026-09-08,2026-09-22,2026-09-21, F6,billing,60d,first-party,CWE-79,pre-release,2026-07-28,2026-09-26,,2026-10-26 F7,accounts,14d,first-party,CWE-352,pre-release,2026-09-25,2026-10-09,, F8,checkout,3d,first-party,CWE-639,production,2026-09-27,2026-09-30,2026-09-29, F9,search,60d,first-party,CWE-79,pre-release,2026-06-01,2026-07-31,, F10,accounts,upgrade,third-party,,pre-release,2026-04-15,2026-10-12,, 14-day-tier time-to-fix sample in days (open = still open on 2026-10-01): 3, 5, 8, 12, 20 open, 30, 41 open, 45 open, 60, 75 open. Proposed dashboard: Findings found this quarter 412 (+38%); Scans run 1,240; MTTR (closed findings) 9 days; SLA met 80%. ``` ### 1. Build the September cohort List the findings due from 2026-09-01 to 2026-09-30 and compute SLA attainment. Worked answer: Seven are due: F1, F2, F3, F4, F5, F6 and F8. F1, F3, F5 and F8 met their dates, so attainment is 4 of 7 (57%). F2 closed late; F4 and F6 are open and overdue, and F6 counts as missed even under its exception. F7 (due 2026-10-09) and F9 (due 2026-07-31) are outside the cohort. ### 2. Explain the 80% Find the calculation behind the dashboard's 80% and say what it hides. Worked answer: On-time closures divided by all closures: 4 of 5. The overdue open findings F4 and F6 drop out of both numbers, and F9, overdue since July, never appears. ### 3. Age the open backlog Bucket the open findings by days since confirmation on 2026-10-01 and list the overdue ones. Worked answer: F7 is 6 days old, F4 26, F6 65, F9 122 and F10 169. Buckets: 0-30: 2; 31-60: 0; 61-90: 1; 91-180: 2; over 180: 0. Overdue: F4, F6 (under exception) and F9. ### 4. Compute escapes and repeats Use first-party findings confirmed in Q3 2026. Worked answer: F1, F4, F5, F6, F7 and F8 qualify; F4, F5 and F8 were found in production, so the escape rate is 3 of 6 (50%). F3 is a third-party advisory and is excluded. F8 repeats CWE-639 in checkout 25 days after F1 was fixed on 2026-09-02. ### 5. Estimate time to remediate Compare the median of closed values with the Kaplan-Meier estimate for the 14-day-tier sample. Worked answer: The six closed values have a median of 10 days. The Kaplan-Meier median is 30 days, and the 90th percentile is not reached because four of ten findings are still open. A mean of closed findings, like the dashboard's 9-day MTTR, ignores open work in the same way. ### 6. Fix the dashboard Replace each proposed panel with one that drives a decision. Worked answer: Replace findings found and scans run with tier-1 coverage and the backlog panel; replace MTTR of closed findings with the Kaplan-Meier median and 90th percentile per tier; replace 80% with cohort attainment shown as 4 of 7, plus the overdue and exception lists. Deliverable: A September 2026 metrics summary that gives each number with its numerator, denominator and definition, plus a revised dashboard layout. Scope: The fixture is small and fictional. With real data, check duplicate handling and re-import dates before trusting any rate, and expect wide uncertainty in quantiles from small samples. Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [OWASP SAMM: Defect Management, Defect Tracking stream (SLAs and exceptions)](https://owaspsamm.org/model/implementation/defect-management/stream-a/) - [OWASP SAMM: Defect Management, Metrics and Feedback stream](https://owaspsamm.org/model/implementation/defect-management/stream-b/) - [OWASP SAMM: Strategy and Metrics practice](https://owaspsamm.org/model/governance/strategy-and-metrics/) - [NIST SP 800-218: SSDF version 1.1 (PO.4, RV.2, RV.3)](https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-218.pdf). Version 1.1 (February 2022). - [OWASP ASVS 5.0.0: V15 Secure Coding and Architecture (15.1.1, 15.2.1)](https://github.com/OWASP/ASVS/blob/v5.0.0_release/5.0/en/0x24-V15-Secure-Coding-and-Architecture.md). Version 5.0.0. - [BSIMM16 report (Black Duck)](https://assets.blackduck.com/image/upload/bsimm-report.pdf). January 2026. - [CISA BOD 26-04: Prioritizing Security Updates Based on Risk](https://www.cisa.gov/news-events/directives/bod-26-04-prioritizing-security-updates-based-risk). 10 June 2026. - [NIST/SEMATECH e-Handbook 8.2.1.5: Empirical model fitting, distribution-free (Kaplan-Meier) approach](https://www.itl.nist.gov/div898/handbook/apr/section2/apr215.htm) - [DefectDojo: SLA configuration](https://docs.defectdojo.com/asset_modelling/os_hierarchy/os__sla_configuration/) - [MITRE CWE-639: Authorization Bypass Through User-Controlled Key](https://cwe.mitre.org/data/definitions/639.html). CWE 4.20.