Automated security tests

Do this

1. List the controls each feature needs

   Start from the feature's security requirements (Security requirements) and abuse cases (Abuse cases): who may call each endpoint, which records and fields each role may read or change, which inputs are refused and which limits apply. OWASP ASVS 5.0.0 gives requirements you can cite by ID, such as v5.0.0-8.2.2 for data-specific access, and the OWASP Web Security Testing Guide has test ideas for each area.

2. Build an authorization matrix

   For each endpoint, list the callers (anonymous, member, admin, admin of another tenant) and the expected result for each (200, 403 or 404), then run one test per cell from that table, as the OWASP Authorization Testing Automation Cheat Sheet describes. Seed two tenants and at least two roles, and change one fact at a time so a failure names the exact rule that broke.

3. Test what must be refused

   Most security requirements say what must not happen, so test the refused case: another tenant's ID returns the same 404 as a missing record, PATCH /users/me with {"role":"admin"} leaves the role unchanged, an order with quantity: -1 is rejected, and confirming an unpaid order fails. NIST's minimum standard for developer verification (NISTIR 8397) lists these negative tests, of what the software should not do, among its techniques.

4. Test limits, errors and logging

   Send 20 redemptions of a single-use coupon at the same instant and expect exactly one success, repeated over several rounds because a race does not show on every run. Force an unexpected exception and expect a generic message with no stack trace (v5.0.0-16.5.1). Request another tenant's record and expect a security log entry naming the user and the resource (v5.0.0-16.3.2).

5. Add a regression test for every fixed vulnerability

   When a penetration test, bug report or scanner finds a real flaw, first write a test that reproduces it and fails, then fix the code and keep the test. NIST SSDF task PW.8.2 gives this as an example: incorporate tests for previously reported vulnerabilities into the test suite so they are not reintroduced. NISTIR 8397 calls them historical test cases. Fix a finding walks through the fix.

6. Run them in CI and make them blocking

   Keep security tests in the same suite and the same required status check as the functional tests, not in a separate job run once before a release. Tag each test with the requirement it proves, for example with pytest's record_property, which adds a property to the test's JUnit XML entry, so a trace check can fail the build when an automated requirement has no passing test.

7. Prove each test can fail

   On a branch, remove the control, such as the tenant filter, and confirm its tests fail; a test that still passes without the control proves nothing. Assert the response body, the stored row or the log entry, not only the status code. Review generated tests as carefully as generated code (AI coding assistants).

Common mistakes

• Testing with one admin account. Most authorization bugs appear only between two tenants or two roles, so a single-user suite passes while the bug ships.
• Testing only the happy path. A test that the owner can read an invoice says nothing about whether anyone else can.
• Asserting only the status code. A 200 can still return fields the caller should never see, and a refused write can still change the stored row; assert the body and the data too.
• Fixing a vulnerability without a regression test, so the next refactor can bring it back and nothing notices until the next penetration test or incident.
• Running security tests in a separate nightly job nobody watches, instead of in the pull request where the author can fix the failure.
• Mocking the layer under test, such as the authorization check, so the test passes whether or not the control exists.

Test cases by control

Starting points drawn from the site's abuse case library, each tied to an OWASP ASVS 5.0.0 requirement. Replace the routes and fields with your own.

Tools and guides

• Workflow: Test an application
• Tool: ASVS 5.0 checklist export
• Abuse case library: tests and controls by feature

  Recurring abuse cases for sign-up and login, password reset, MFA enrolment, invites, file upload, payments and coupons, webhooks, search, export, admin and AI chat or agents: the attacker goal, preconditions, a concrete test and the control, each linked to CWE and CAPEC.

• Write testable security requirements and a security Definition of Done

  Derive security requirements from OWASP ASVS 5.0, abuse cases and the threat model, write them as acceptance criteria a test can fail, trace each one to a test or evidence, and close stories against a short security Definition of Done.

• Enforce authorization for every object action

  Authorize the authenticated subject, requested operation, and specific resource together; a valid login or unpredictable identifier is insufficient.

• Authorize functions and fields as well as objects

  Prevent mass assignment, sensitive-field exposure and administrative actions that a valid object lookup alone cannot protect.

Standards

NIST SSDF 1.1

PW.8 Test Executable Code to Identify Vulnerabilities and Verify Compliance with Security Requirements

OWASP SAMM 2

Requirements-driven TestingSecurity Testing

Sources

1. NIST SP 800-218: Secure Software Development Framework (SSDF) Version 1.1. February 2022.
2. NIST IR 8397: Guidelines on Minimum Standards for Developer Verification of Software. July 2021, with October 2021 errata.
3. OWASP SAMM: Requirements-driven Testing
4. OWASP ASVS 5.0.0: V8 Authorization. Version 5.0.0.
5. OWASP ASVS 5.0.0: V2 Validation and Business Logic. Version 5.0.0.
6. OWASP ASVS 5.0.0: V15 Secure Coding and Architecture. Version 5.0.0.
7. OWASP ASVS 5.0.0: V16 Security Logging and Error Handling. Version 5.0.0.
8. OWASP Authorization Testing Automation Cheat Sheet
9. OWASP Web Security Testing Guide. v4.2, the stable release; v5.0 in development.
10. pytest: record_property and JUnit XML output