Choose and evaluate application security testing tools

Each category sees one representation of the software: source code, a running application, an instrumented runtime, a dependency list, infrastructure definitions, an image or an API schema. That determines what it can find. None of them knows your authorization rules or business invariants, so missing or broken access control (CWE-862, CWE-863, CWE-639) and logic flaws stay with threat modeling, code review and tests written from abuse cases, such as those in /domains/appsec/threat-modeling/abuse-case-library/.

NIST SP 800-53 Rev. 5 names the testing techniques as control enhancements: static code analysis (SA-11(1)), dynamic code analysis (SA-11(8)) and interactive application security testing (SA-11(9)), with runtime application self-protection (SI-7(17)) as a protection control rather than a test.

To show what a measured comparison looks like, we built a small Flask application on 2026-10-10 with 15 seeded problems (S1 to S15) and 10 safe twins (N1 to N10). The twins do the same job safely, so a tool that flags them is producing false positives. We ran each tool with its default configuration, as a team would on first install. Treat the results as an example of the method, not a ranking: one small Python app says little about other languages or about the same tools tuned.

Semgrep reported 21 results and Bandit 11, including two import warnings outside any seeded item. Scored against the 12 code-level items they could see, Semgrep found 8 with 2 false positives (Youden's index 47) and Bandit found 6 with 1 (Youden's index 40). Bandit's run covered only the application code, not the Dockerfile. Restricting Semgrep to ERROR severity, a common gate, kept 11 results: the SQL, command injection and SSRF findings and both N6 false positives, while dropping the XSS, pickle and MD5 findings.

• Rule counts mislead. In the p/default ruleset served that day, 6 Python rules carry CWE-22 and 2 carry CWE-601, and none matched S3 or S7. Measure detection on patterns from your own code.
• Safe twins expose noise early. N6 shows that an allowlist check written in your style may not be recognized as a sanitizer; tune with a custom sanitizer rule rather than suppressing the SSRF rule.
• CWE tags are inconsistent. Two Semgrep rules that flagged S1, both named tainted-sql-string, were tagged CWE-915 and CWE-704 instead of CWE-89, so map results to the seeded CWE before computing per-CWE rates.
• Scope changes results. Bandit read only Python and could not see S14; gitleaks in directory mode missed S15; Semgrep's default ruleset ignored the Terraform file.
• Resolution changes SCA results. OSV-Scanner's own resolution reported idna 2.9.0 and urllib3 1.26.9, while pip resolved idna 2.10 and urllib3 1.26.20 for the same file on the same day. Scan a lockfile that records what you install.
• Images add another layer. Trivy found 166 Debian package findings in the built image, only 2 with a fix, and 37 Python package findings, 10 of them fixable at high or critical severity. That gap is why scan gates usually block only on fixable findings.

• Build the corpus from your stack: your languages, frameworks and the patterns behind your past incidents and pentest findings. Give every seeded flaw a safe twin written in your team's style.
• Freeze the answer key before running anything: file, line range, CWE and whether each item is vulnerable.
• Run each candidate on the same commit with defaults, then with the configuration you would actually ship, and keep the commands and versions.
• Map every result to a seeded item or mark it outside. Compute true and false positive rates per CWE and in total, and the results per true positive.
• Triage a sample of results from two or three real repositories with the engineers who will own the tool, and time it: minutes per result times results per real issue is the running cost.
• Add breadth from public suites: OWASP Benchmark for Java v1.2 has 2,740 test cases across 11 CWEs (1,415 real vulnerabilities, 1,325 false-positive traps), OWASP Benchmark for Python v0.1 has 1,230 across 14 CWEs, and NIST's Juliet suites cover 118 CWEs for C and C++ and 112 for Java.

Most scanners write SARIF 2.1.0, the OASIS format that code scanning services import, so one script can score them all. This one counts a seeded item as found when any result falls inside its line range, and reports found and missed items, false positives on safe twins, Youden's index and results per true positive. Its output for our run matched the table above. For DAST, IAST and API scanners, key the answer to routes and parameters instead of lines.

OWASP Benchmark scores the same way: its Benchmark Accuracy Score is the true positive rate minus the false positive rate, scaled to 0 to 100, and can be negative when a tool flags more traps than real flaws.

// score-sarif.mjs: score SARIF results against an answer key of seeded flaws and safe twins.
// node score-sarif.mjs answer-key.json results.sarif [more.sarif ...]
// answer-key.json: [{"id": "S1", "cwe": "CWE-89", "file": "app/routes.py", "lines": [25, 31], "vulnerable": true}, ...]
import {readFileSync} from 'node:fs';

const [keyFile, ...sarifFiles] = process.argv.slice(2);
const key = JSON.parse(readFileSync(keyFile, 'utf8'));
const normalize = uri => uri.replace(/^file:\/\//, '').replace(/^(\.\/|\/src\/|seeded\/)/, '');

for (const file of sarifFiles) {
  const sarif = JSON.parse(readFileSync(file, 'utf8'));
  const results = sarif.runs.flatMap(run => run.results || []);
  const hits = new Map(key.map(item => [item.id, 0]));
  let outside = 0;
  for (const result of results) {
    const where = result.locations?.[0]?.physicalLocation;
    const uri = normalize(where?.artifactLocation?.uri || '');
    const line = where?.region?.startLine || 0;
    const item = key.find(k => k.file === uri && line >= k.lines[0] && line <= k.lines[1]);
    if (item) hits.set(item.id, hits.get(item.id) + 1);
    else outside++;
  }
  const flagged = item => hits.get(item.id) > 0;
  const vulns = key.filter(k => k.vulnerable);
  const twins = key.filter(k => !k.vulnerable);
  const tp = vulns.filter(flagged), fp = twins.filter(flagged);
  const tpr = tp.length / vulns.length, fpr = twins.length ? fp.length / twins.length : 0;
  console.log(`${file}: ${results.length} results, ${outside} outside any seeded item`);
  console.log(`  found ${tp.map(k => k.id).join(' ') || '-'}; missed ${vulns.filter(k => !flagged(k)).map(k => k.id).join(' ') || '-'}`);
  console.log(`  false positives on ${fp.map(k => k.id).join(' ') || '-'}`);
  console.log(`  TPR ${(tpr * 100).toFixed(0)}%  FPR ${(fpr * 100).toFixed(0)}%  Youden ${((tpr - fpr) * 100).toFixed(0)}  results per true positive ${(results.length / Math.max(tp.length, 1)).toFixed(1)}`);
}

A twin should differ from its flaw only in the property that makes it safe, so a tool must understand that property to stay quiet. Write twins the way your team actually writes safe code, including helper functions and allowlists, because those are what a tool must recognize. Never deploy the corpus, and keep fake secrets in a format that cannot be valid.

# app/routes.py (excerpt): every seeded flaw (S) has a safe twin (N) that should not be flagged.
# S3 CWE-22: path joined from request data
@app.get("/s3/report")
def s3_report():
    return send_file(os.path.join(REPORTS, request.args.get("name", "")))


# N3: send_from_directory rejects paths that leave the directory
@app.get("/n3/report")
def n3_report():
    return send_from_directory(REPORTS, request.args.get("name", ""))


# S6 CWE-918: server-side fetch of a request-supplied URL
@app.get("/s6/preview")
def s6_preview():
    return requests.get(request.args.get("url", ""), timeout=5).text


# N6: host allowlist checked before the fetch
@app.get("/n6/preview")
def n6_preview():
    url = request.args.get("url", "")
    parsed = urlparse(url)
    if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS:
        return "forbidden", 403
    return requests.get(url, timeout=5, allow_redirects=False).text

Adopt in the order that buys the most risk reduction for the least tuning, and put each tool where its input appears: /domains/appsec/testing/devsecops-pipeline/ maps tools to stages, and the reference pipeline project on /labs/ shows them pinned in GitHub Actions and GitLab CI.

• Choosing on the vendor's demo application instead of your code.
• Comparing tools on findings found without counting false positives and duplicates.
• Reading a clean result as proof of security; a tool reports only what its rules and inputs let it see.
• Enabling every rule at blocking severity on day one, which trains developers to bypass the gate.
• Ignoring rule licensing: Semgrep CE is LGPL-2.1, but Semgrep-maintained registry rules carry their own license terms, and CodeQL is free only for open-source code, and for automated CI analysis only when that code is hosted on GitHub.com.
• Buying ASPM before the scanners feeding it are tuned; aggregation of noise is still noise.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. OWASP Benchmark project
2. OWASP Benchmark for Java (v1.2 test cases and expected results)
3. OWASP Benchmark for Python
4. NIST SARD: Juliet C/C++ 1.3
5. NIST SARD: Juliet Java 1.3
6. NIST IR 8397: minimum standards for developer verification of software. October 2021.
7. NIST SP 800-53 Rev. 5 control catalog (SA-11, SI-7). Release 5.1.1.
8. OASIS SARIF 2.1.0
9. ZAP API scan
10. Schemathesis documentation
11. Semgrep licensing
12. OWASP DefectDojo

Reference information

Tags

Secrets managementDependency managementInfrastructure as codeKubernetes and containersAPI securitySecurity testingVulnerability managementAppSec programs

SDLC phase

TestingMaintenanceSecurity program

Role

AppSec engineersPlatform engineers and SREsSecurity leaders

Download this guide as Markdown

Continue exploring