Write testable security requirements and a security Definition of Done

A requirement is testable when someone other than its author can decide, from the running system or its records, whether it holds. OWASP SAMM's first Security Requirements activity puts it this way: a requirement states the objective, such as protecting personal data in transit, and not the measure that achieves it. It also warns against requirements so general that they add nothing, such as "protect against the OWASP Top 10".

Write each requirement as one sentence with a subject, an outcome and a boundary. Give it an ID, a source (ASVS, an abuse case, a threat, a law or a past finding) and a verification method: an automated test, a review or a manual test with evidence. If you cannot say how it will be verified, it is not finished.

Use four sources and record which one produced each requirement, so a reviewer can see why it exists. SAMM's second-level activity asks for the same breadth: policies and legislation, known problems in the application, and feedback from metrics and incidents, written in the notation the team already uses for other requirements.

• A baseline from ASVS 5.0 at the application's level, filtered to the chapters the feature touches. It catches the controls every feature of that kind needs.
• Abuse cases for the feature: what a hostile user, a curious customer or a compromised integration would try. The abuse case library at /domains/appsec/threat-modeling/abuse-case-library/ has tested examples by feature.
• The threat model: each threat you decide to mitigate becomes a requirement with a test. The guide at /domains/appsec/threat-modeling/threat-modeling-methods/ shows how, and /domains/appsec/threat-modeling/threat-pattern-library/ lists recurring patterns.
• Obligations and history: contract clauses, PCI DSS Requirement 6 for payment code (/frameworks/pci-dss-requirement-6/), regulation, and the CWE classes that keep recurring in your own findings.

ASVS 5.0.0 has three levels. Level 1 holds about 20% of the requirements and is meant as a low-barrier starting point. Level 2 adds about half of the requirements, so complying with it means meeting about 70% of the standard, and ASVS says most applications should strive for it. Level 3 adds the final 30% for applications that must show the highest assurance. ASVS leaves the choice to the organization's risk analysis; its own example is a startup holding little sensitive data starting at level 1, and a bank struggling to justify less than level 3 for online banking.

Record the level per application, not per team, and revisit it when the data or exposure changes. ASVS encourages organizations to fork the standard and drop sections that do not apply, such as GraphQL or WebSockets, as long as an ID keeps its meaning. A fork should start from level 1 and add requirements by risk.

Start from what the feature does, not from the chapter list. Map each capability to the chapters that govern it, then keep only the requirements at or below the application's level. Most stories touch two to four chapters. Cite every requirement with the version prefix, because ASVS says IDs may change between versions and an unversioned ID refers to the latest release.

Prefer requirements that state a verifiable behavior, such as v5.0.0-8.2.2 (data-specific access restricted to consumers with explicit permission), over documentation requirements such as v5.0.0-8.1.1, which belong to the design record rather than to a single story.

Write criteria in the format the team already uses. Given/When/Then works well because the "Then" forces an observable result. Four rules keep them useful: one behavior per scenario; always include the case that must be refused; state what the caller can observe (status, body, log entry), not the implementation; and tag the scenario with the requirement and ASVS IDs so tests and reports can be traced.

The example covers story EXP-14 at ASVS level 2. Note the second scenario: it does not only check that another tenant's invoice is refused, it checks that the refusal looks identical to a missing invoice, so IDs cannot be probed, as /domains/appsec/authorization/object-level-authorization/ explains.

# Story EXP-14: download an invoice. ASVS 5.0.0 level 2.
Feature: Invoice download stays inside the caller's tenant

  Background:
    Given Ana is signed in as a member of tenant "red"
    And invoice "inv-1" belongs to tenant "red"
    And invoice "inv-2" belongs to tenant "blue"

  @SR-1 @v5.0.0-8.2.2
  Scenario: A member reads an invoice in their own tenant
    When Ana requests invoice "inv-1"
    Then the response status is 200
    And the response contains the invoice total

  @SR-1 @v5.0.0-8.2.2
  Scenario: Another tenant's invoice looks the same as a missing one
    When Ana requests invoice "inv-2"
    And Ana requests invoice "inv-999"
    Then both responses have status 404 and identical bodies

  @SR-2 @v5.0.0-8.3.1
  Scenario: A tenant named in the request is ignored
    When Ana requests invoice "inv-2" with the query parameter tenant=blue
    Then the response status is 404

  @SR-3 @v5.0.0-16.3.2
  Scenario: A denied read is logged without the invoice contents
    When Ana requests invoice "inv-2"
    Then the security log has an "authz_denied" entry naming user "ana" and resource "invoice:inv-2"

Each scenario becomes a test that fails if the control is removed. The pytest tests below run against a small Flask service whose handler loads the invoice and compares its tenant with the tenant of the authenticated user, returning 404 in both the missing and the foreign case. Each test records the requirement it proves with pytest's record_property, which writes a property into the JUnit XML report.

Run them with pytest -o junit_family=xunit1 --junitxml=junit.xml; pytest warns that record_property is not compatible with the default xunit2 report family. Removing the tenant comparison from the handler makes three of the four tests fail, which is the point: a requirement test that still passes without the control proves nothing.

import logging

import pytest

from app import app

ANA = {"Authorization": "Bearer token-ana"}  # tenant red


@pytest.fixture
def client():
    return app.test_client()


def test_owner_can_read_own_invoice(client, record_property):
    record_property("requirement", "SR-1")
    response = client.get("/invoices/inv-1", headers=ANA)
    assert response.status_code == 200
    assert response.get_json()["total"] == 40


def test_other_tenant_invoice_looks_missing(client, record_property):
    record_property("requirement", "SR-1")  # v5.0.0-8.2.2
    other = client.get("/invoices/inv-2", headers=ANA)
    missing = client.get("/invoices/inv-999", headers=ANA)
    assert other.status_code == missing.status_code == 404
    assert other.data == missing.data


def test_client_supplied_tenant_is_ignored(client, record_property):
    record_property("requirement", "SR-2")  # v5.0.0-8.3.1
    response = client.get("/invoices/inv-2?tenant=blue", headers=ANA)
    assert response.status_code == 404


def test_denied_access_is_logged(client, caplog, record_property):
    record_property("requirement", "SR-3")  # v5.0.0-16.3.2
    with caplog.at_level(logging.WARNING, logger="security"):
        client.get("/invoices/inv-2", headers=ANA)
    assert "authz_denied user=ana resource=invoice:inv-2" in caplog.text

Keep the requirements next to the code, in a file that both people and CI can read. Each entry carries its ID, its text, its source, the ASVS IDs it covers and how it is verified. Requirements that cannot be automated, such as a rate limit proven by a load test, name the evidence instead. NIST SSDF task PW.1.2 asks you to track and maintain the software's security requirements, risks and design decisions; a version-controlled file does that and shows every change in review.

# Security requirements for story EXP-14 "Download an invoice".
# Each one names its source, its ASVS 5.0.0 requirement and how it is verified.
feature: invoice-download
asvs_level: 2
requirements:
  - id: SR-1
    text: A user can read an invoice only if it belongs to their tenant; any other invoice ID gets the same 404 as a missing one.
    source: [abuse-case AC-EXP-1 "read another tenant's invoice by changing the ID"]
    asvs: [v5.0.0-8.2.2]
    verification: automated
  - id: SR-2
    text: The tenant used for the check comes from the authenticated session, never from a request parameter, header or body.
    source: [threat-model T3]
    asvs: [v5.0.0-8.3.1]
    verification: automated
  - id: SR-3
    text: Every denied read is logged with user, resource and UTC time, without the invoice contents.
    source: [incident-response need]
    asvs: [v5.0.0-16.3.2]
    verification: automated
  - id: SR-4
    text: Bulk invoice export is limited to 20 requests per user per minute.
    source: [abuse-case AC-EXP-3 "scrape every invoice through the export API"]
    asvs: [v5.0.0-2.4.1]
    verification: manual
    evidence: Load-test report attached to the release ticket.

Add a CI step after the tests that compares the requirements file with the JUnit report. It fails when an automated requirement has no test recording it, when any of its tests failed, or when a manual requirement has no evidence. With the service above it prints 4/4 requirements traced; with the tenant check removed it lists SR-1, SR-2 and SR-3 and exits with status 1.

The gate checks that tests exist and pass; it cannot judge whether a test is meaningful. Review new requirement tests the way you review the control itself, and break the control once on a branch to see the test fail.

"""Fail the build when an automated security requirement has no passing test.

Usage: python check_trace.py requirements.yaml junit.xml
"""
import sys
import xml.etree.ElementTree as ET

import yaml


def main(requirements_path, junit_path):
    spec = yaml.safe_load(open(requirements_path, encoding="utf-8"))
    passed, failed = {}, {}
    for case in ET.parse(junit_path).iter("testcase"):
        ok = not any(case.find(tag) is not None for tag in ("failure", "error", "skipped"))
        for prop in case.iter("property"):
            if prop.get("name") == "requirement":
                (passed if ok else failed).setdefault(prop.get("value"), []).append(case.get("name"))
    problems = []
    for req in spec["requirements"]:
        rid = req["id"]
        if req["verification"] == "automated":
            if rid in failed:
                problems.append(f"{rid}: failing test {failed[rid]}")
            elif rid not in passed:
                problems.append(f"{rid}: no test records this requirement")
        elif not req.get("evidence"):
            problems.append(f"{rid}: manual requirement without evidence")
    for line in problems:
        print("FAIL", line)
    print(f"{len(spec['requirements']) - len(problems)}/{len(spec['requirements'])} requirements traced")
    return 1 if problems else 0


if __name__ == "__main__":
    sys.exit(main(*sys.argv[1:3]))

A Definition of Done is the checklist every item must pass before it counts as finished. Keep the security part short and binary, and split it by level: what a story needs before it closes, what a feature needs before customers see it, and what a release needs before it ships. Items that apply to everything, such as "no new SAST finding at blocking severity", belong here rather than in every story's requirements.

Make each item checkable by someone outside the team, and automate the ones you can. The trace gate covers the first three story items; scanners cover the SAST and dependency items; the rest are short review questions in the pull request or release template.

# Security Definition of Done. Every item is a yes/no check with an owner.
story:   # before the story moves to Done
  - Security requirements are listed on the story with an ASVS 5.0.0 ID or "local" and a source.
  - Each automated requirement has a test that fails when the control is removed.
  - The security trace check passes (no automated requirement without a passing test).
  - New endpoints and fields are covered by the authorization tests for their resource.
  - No new SAST or secret-scanning finding at or above the team's blocking severity.
  - Abuse cases raised in refinement are either tested or recorded as accepted risk by the owner.
feature: # before the feature flag is turned on for customers
  - The threat model for the feature is updated and its open threats have owners.
  - Manual requirements have evidence attached (test report, review record or screenshot).
  - Security events the feature adds are logged and reach the log pipeline.
  - Dependencies added by the feature are in the SBOM and have no known exploited vulnerability.
release: # before a release is published
  - Every requirement at the application's ASVS level is traced or has a dated exception.
  - Exceptions have an owner and an expiry date.
  - Release notes or an advisory describe any security fix customers must act on.

The practice above satisfies several frameworks without extra paperwork. Cite the framework edition when an assessor asks.

• Copying the whole ASVS into every story. Select by feature and level; a 300-line checklist gets ticked, not read.
• Writing mechanisms instead of outcomes. "Use the auth middleware" passes even when a new route forgets it; "every route under /invoices checks the tenant" fails.
• Testing only the happy path. Most security requirements are about what must not happen, so the refused case is the test that matters.
• Counting "security reviewed" as evidence. Name the reviewer, the date and what was checked, or automate it.
• Letting requirements drift from code. Keep them in the repository and review changes to them like code.
• Treating the Definition of Done as the requirements. The DoD applies to everything; requirements are specific to the feature and its risks.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP ASVS 5.0.0: Using the ASVS (levels and requirement identifiers). Version 5.0.0.
2. OWASP ASVS 5.0.0: V8 Authorization. Version 5.0.0.
3. OWASP ASVS 5.0.0: V6 Authentication. Version 5.0.0.
4. OWASP ASVS 5.0.0: V16 Security Logging and Error Handling. Version 5.0.0.
5. OWASP SAMM: Security Requirements, stream A (Software Requirements)
6. NIST SP 800-218: Secure Software Development Framework 1.1. Version 1.1 (February 2022).
7. OWASP Abuse Case Cheat Sheet
8. Cucumber: Gherkin reference
9. pytest: record_property and JUnit XML output

Reference information

Tags

Threat modelingSecurity testingSecure development lifecycleCompliance and assurance evidence

SDLC phase

PlanningDesignTestingSecurity program

Role

DevelopersAppSec engineers

Download this guide as Markdown

Continue exploring