--- key: "appsec-tool-categories-and-evaluation" title: "Choose and evaluate application security testing tools" url: "https://softwaresecurity.com/domains/appsec/testing/appsec-tool-categories-and-evaluation/" api: "https://softwaresecurity.com/api/v1/articles/appsec-tool-categories-and-evaluation.json" domain: "appsec" topic: "Application security" type: "Guide" identifiers: ["A03:2025","A05:2025","V15.1","V15.2"] tags: ["Secrets management","Dependency management","Infrastructure as code","Kubernetes and containers","API security","Security testing","Vulnerability management","AppSec programs"] stages: ["test","respond","govern"] roles: ["appsec-engineer","platform-sre","security-leader"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Choose and evaluate application security testing tools What SAST, DAST, IAST, SCA, secrets, IaC, container and API scanners and ASPM each find and miss, where each runs, how to evaluate them on a seeded repository with per-CWE scoring and triage cost, and which to adopt first. Canonical URL: https://softwaresecurity.com/domains/appsec/testing/appsec-tool-categories-and-evaluation/ ## In short Each category looks at one representation of your software. SAST reads source code and finds injection-style data flows. DAST attacks the running app over HTTP. IAST instruments the app while tests run. SCA matches your dependencies to known advisories. Secrets scanners look for credentials, IaC scanners check infrastructure definitions, container scanners check image packages, and API scanners generate requests from your API schema. ASPM aggregates the other tools' findings and detects nothing on its own. Every category misses whole classes of flaws, authorization above all. Choose by measuring, not by feature lists: seed a repository from your own stack with known flaws and safe twins, run each candidate, and score true and false positive rates per CWE and the triage time per real issue. For most teams the order of adoption is secrets and SCA, then SAST on pull requests, then DAST or API testing against a test deployment. - On our seeded Flask app, Semgrep CE's default rules found 8 of 12 seeded flaws and flagged 2 of 10 safe twins; Bandit found 6 and flagged 1. Neither found the path traversal, the open redirect or the missing ownership check. - A rule count is not coverage: the default Semgrep ruleset has 6 Python rules mapped to CWE-22 and 2 to CWE-601, and none fired on our `send_file` and redirect flaws. - Secrets scanning must read history: gitleaks found one token in the working tree and two in git mode, including one deleted in a later commit. - SCA results depend on what the scanner resolves: OSV-Scanner reported 22 advisories for 5 pinned packages and 34 with transitive resolution, using versions that differed from what pip installed. - Score with both rates: OWASP Benchmark uses true positive rate minus false positive rate (Youden's index), so a tool that flags everything scores 0. - ASPM and DefectDojo-style aggregation reduce duplicates and route findings to owners; they cannot find what the underlying scanners miss. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Evaluation corpus | Seeded flaws from your own stack and past incidents, each with a safe twin, frozen in an answer key before any tool runs | — | — | | Scoring | Per CWE: true positive rate, false positive rate and their difference (Youden's index) | — | https://owasp.org/projects/benchmark | | Triage cost | Results per true positive, times measured minutes per result | — | — | | Result format | SARIF 2.1.0 with stable rule IDs and CWE tags | — | https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html | | First tools to adopt | Secrets scanning with history and push protection; SCA on lockfiles | Highest precision for the least tuning; SAST next. | — | ## Verify it 1. Every candidate ran on the same commit of the seeded corpus with its configuration recorded, and the answer key was written before the first run. 2. Each reported result is mapped to a seeded item or marked outside, and per-CWE true and false positive rates are recorded for each tool. 3. A sample of results from two or three real repositories was triaged by the team that will own the tool, with minutes per result recorded. 4. The chosen tool blocks a pull request that adds one of the seeded flaws, and stays quiet on one that adds its safe twin. 5. The decision record states which CWE classes no adopted tool covers and which test or review covers them instead. ## What you will learn - Describe what each tool category can and cannot detect, by CWE, and where it runs. - Build a seeded corpus with safe twins and score tools per CWE with true and false positive rates. - Estimate the triage cost of a tool from results per real finding. - Choose which categories to adopt first for a given application and team. Audience: AppSec engineers, engineering managers and platform teams who choose, buy or roll out security scanners Before you start: - Know your languages, frameworks, package managers and deployment targets. - Read SARIF or a scanner's JSON output. Time: About 32 minutes: 10 to read, 3 on the code and 19 for the exercise ## What each category finds and misses Each category sees one representation of the software: source code, a running application, an instrumented runtime, a dependency list, infrastructure definitions, an image or an API schema. That determines what it can find. None of them knows your authorization rules or business invariants, so missing or broken access control (CWE-862, CWE-863, CWE-639) and logic flaws stay with threat modeling, code review and tests written from abuse cases, such as those in /domains/appsec/threat-modeling/abuse-case-library/. NIST SP 800-53 Rev. 5 names the testing techniques as control enhancements: static code analysis (SA-11(1)), dynamic code analysis (SA-11(8)) and interactive application security testing (SA-11(9)), with runtime application self-protection (SI-7(17)) as a protection control rather than a test. | Category | Finds well | Misses | Where it runs | Open-source examples | | --- | --- | --- | --- | --- | | SAST | Data flows from input to dangerous sinks: SQL, command and template injection, XSS, unsafe deserialization, weak crypto calls | Authorization and logic flaws, framework behavior its rules do not model, configuration | Editor and pull request; full scan nightly | Semgrep CE, Opengrep, CodeQL, Bandit, gosec, SpotBugs with Find Security Bugs | | DAST | What the running app does: missing headers and cookie flags, reflected XSS, some injection, TLS problems | Pages it cannot reach or log in to, logic flaws, root cause in code | Test or staging deployment | ZAP, Nuclei | | IAST | Vulnerable data flows observed during tests, with the request and stack that triggered them | Code your tests never execute; unsupported runtimes | Test environment, as an agent in the app | Mostly commercial agents for specific runtimes | | SCA | Known-vulnerable and malicious dependencies, license terms | Flaws with no advisory, vendored or copied code, whether vulnerable code is reachable (unless it does call analysis) | Pull request on lockfiles; nightly for new advisories | OSV-Scanner, Trivy, Grype, OWASP Dependency-Check, Dependency-Track | | Secrets | Credentials in code, config and git history | Custom token formats, encoded or split values, secrets outside the repository | Pre-commit, pull request, full history, server-side push protection | gitleaks, TruffleHog | | IaC | Public storage, open ports, missing encryption and logging in Terraform, Kubernetes manifests and Dockerfiles | Drift after deployment, effective permissions across policies | Pull request | Checkov, Trivy, KICS | | Container | Vulnerable OS and language packages in the image, secrets in layers, root user | Flaws in your own code, settings applied at deploy time | After build, and continuously in the registry | Trivy, Grype | | API | Server errors, schema violations, crashes and some authorization flaws, from requests generated from OpenAPI or GraphQL schemas | Endpoints missing from the schema; cross-user access unless given two identities | Test deployment | Schemathesis, ZAP API scan | | ASPM | Duplicates, ownership, risk ranking and trends across all of the above | Anything its feeding scanners miss; it detects nothing | Central service fed by CI | OWASP DefectDojo | ## What the tools found in a seeded repository To show what a measured comparison looks like, we built a small Flask application on 2026-10-10 with 15 seeded problems (S1 to S15) and 10 safe twins (N1 to N10). The twins do the same job safely, so a tool that flags them is producing false positives. We ran each tool with its default configuration, as a team would on first install. Treat the results as an example of the method, not a ranking: one small Python app says little about other languages or about the same tools tuned. Semgrep reported 21 results and Bandit 11, including two import warnings outside any seeded item. Scored against the 12 code-level items they could see, Semgrep found 8 with 2 false positives (Youden's index 47) and Bandit found 6 with 1 (Youden's index 40). Bandit's run covered only the application code, not the Dockerfile. Restricting Semgrep to ERROR severity, a common gate, kept 11 results: the SQL, command injection and SSRF findings and both N6 false positives, while dropping the XSS, pickle and MD5 findings. | Seeded item | Semgrep CE 1.180.0 | Bandit 1.9.4 | Other tools | | --- | --- | --- | --- | | S1 SQL built with an f-string (CWE-89) | Found, 5 results | Found | - | | S2 `subprocess.run(..., shell=True)` with request data (CWE-78) | Found, 3 results | Found | - | | S3 `send_file(os.path.join(...))` with request data (CWE-22) | Missed | Missed | - | | S4 request data in an HTML f-string (CWE-79) | Found, 3 results | Missed | - | | S5 `pickle.loads` of the request body (CWE-502) | Found | Found | - | | S6 `requests.get` of a request-supplied URL (CWE-918) | Found, 2 results | Missed | - | | S7 redirect to the `next` parameter (CWE-601) | Missed | Missed | - | | S8 unsalted MD5 password hash (CWE-328) | Found | Found | - | | S9 invoice returned without an owner check (CWE-639) | Missed | Missed | No category can see the rule | | S10 GitHub token in source (CWE-798) | Missed | Found, by variable name | gitleaks and Trivy found it | | S11 `app.run(debug=True)` (CWE-489) | Found, 2 results | Found | - | | S12 five pinned packages with advisories (CWE-1395) | - | - | OSV-Scanner: 22 advisories, 34 with resolution; Trivy fs: 22 | | S13 SSH open to `0.0.0.0/0` in Terraform | Not covered by `p/default` | - | Checkov `CKV_AWS_24`, Trivy `AWS-0107` | | S14 Dockerfile without `USER` (CWE-250) | Found | Not in scope | Checkov `CKV_DOCKER_3`, Trivy `DS-0002` | | S15 token committed, then deleted (CWE-798) | - | - | gitleaks in git mode only | | False positives on safe twins | N4 `render_template_string` with a constant template; N6 allowlisted URL, 2 results | N2 argument-list subprocess, 2 low-severity results | Checkov `CKV2_AWS_5` on both security groups, unattached in the fixture | ## What the run teaches about evaluating tools - Rule counts mislead. In the p/default ruleset served that day, 6 Python rules carry CWE-22 and 2 carry CWE-601, and none matched S3 or S7. Measure detection on patterns from your own code. - Safe twins expose noise early. N6 shows that an allowlist check written in your style may not be recognized as a sanitizer; tune with a custom sanitizer rule rather than suppressing the SSRF rule. - CWE tags are inconsistent. Two Semgrep rules that flagged S1, both named `tainted-sql-string`, were tagged CWE-915 and CWE-704 instead of CWE-89, so map results to the seeded CWE before computing per-CWE rates. - Scope changes results. Bandit read only Python and could not see S14; gitleaks in directory mode missed S15; Semgrep's default ruleset ignored the Terraform file. - Resolution changes SCA results. OSV-Scanner's own resolution reported idna 2.9.0 and urllib3 1.26.9, while pip resolved idna 2.10 and urllib3 1.26.20 for the same file on the same day. Scan a lockfile that records what you install. - Images add another layer. Trivy found 166 Debian package findings in the built image, only 2 with a fix, and 37 Python package findings, 10 of them fixable at high or critical severity. That gap is why scan gates usually block only on fixable findings. ## Run your own evaluation in six steps - Build the corpus from your stack: your languages, frameworks and the patterns behind your past incidents and pentest findings. Give every seeded flaw a safe twin written in your team's style. - Freeze the answer key before running anything: file, line range, CWE and whether each item is vulnerable. - Run each candidate on the same commit with defaults, then with the configuration you would actually ship, and keep the commands and versions. - Map every result to a seeded item or mark it outside. Compute true and false positive rates per CWE and in total, and the results per true positive. - Triage a sample of results from two or three real repositories with the engineers who will own the tool, and time it: minutes per result times results per real issue is the running cost. - Add breadth from public suites: OWASP Benchmark for Java v1.2 has 2,740 test cases across 11 CWEs (1,415 real vulnerabilities, 1,325 false-positive traps), OWASP Benchmark for Python v0.1 has 1,230 across 14 CWEs, and NIST's Juliet suites cover 118 CWEs for C and C++ and 112 for Java. ## Score SARIF results against the answer key Most scanners write SARIF 2.1.0, the OASIS format that code scanning services import, so one script can score them all. This one counts a seeded item as found when any result falls inside its line range, and reports found and missed items, false positives on safe twins, Youden's index and results per true positive. Its output for our run matched the table above. For DAST, IAST and API scanners, key the answer to routes and parameters instead of lines. OWASP Benchmark scores the same way: its Benchmark Accuracy Score is the true positive rate minus the false positive rate, scaled to 0 to 100, and can be negative when a tool flags more traps than real flaws. ```javascript // score-sarif.mjs: score SARIF results against an answer key of seeded flaws and safe twins. // node score-sarif.mjs answer-key.json results.sarif [more.sarif ...] // answer-key.json: [{"id": "S1", "cwe": "CWE-89", "file": "app/routes.py", "lines": [25, 31], "vulnerable": true}, ...] import {readFileSync} from 'node:fs'; const [keyFile, ...sarifFiles] = process.argv.slice(2); const key = JSON.parse(readFileSync(keyFile, 'utf8')); const normalize = uri => uri.replace(/^file:\/\//, '').replace(/^(\.\/|\/src\/|seeded\/)/, ''); for (const file of sarifFiles) { const sarif = JSON.parse(readFileSync(file, 'utf8')); const results = sarif.runs.flatMap(run => run.results || []); const hits = new Map(key.map(item => [item.id, 0])); let outside = 0; for (const result of results) { const where = result.locations?.[0]?.physicalLocation; const uri = normalize(where?.artifactLocation?.uri || ''); const line = where?.region?.startLine || 0; const item = key.find(k => k.file === uri && line >= k.lines[0] && line <= k.lines[1]); if (item) hits.set(item.id, hits.get(item.id) + 1); else outside++; } const flagged = item => hits.get(item.id) > 0; const vulns = key.filter(k => k.vulnerable); const twins = key.filter(k => !k.vulnerable); const tp = vulns.filter(flagged), fp = twins.filter(flagged); const tpr = tp.length / vulns.length, fpr = twins.length ? fp.length / twins.length : 0; console.log(`${file}: ${results.length} results, ${outside} outside any seeded item`); console.log(` found ${tp.map(k => k.id).join(' ') || '-'}; missed ${vulns.filter(k => !flagged(k)).map(k => k.id).join(' ') || '-'}`); console.log(` false positives on ${fp.map(k => k.id).join(' ') || '-'}`); console.log(` TPR ${(tpr * 100).toFixed(0)}% FPR ${(fpr * 100).toFixed(0)}% Youden ${((tpr - fpr) * 100).toFixed(0)} results per true positive ${(results.length / Math.max(tp.length, 1)).toFixed(1)}`); } ``` ## Seed flaws with safe twins A twin should differ from its flaw only in the property that makes it safe, so a tool must understand that property to stay quiet. Write twins the way your team actually writes safe code, including helper functions and allowlists, because those are what a tool must recognize. Never deploy the corpus, and keep fake secrets in a format that cannot be valid. ```python # app/routes.py (excerpt): every seeded flaw (S) has a safe twin (N) that should not be flagged. # S3 CWE-22: path joined from request data @app.get("/s3/report") def s3_report(): return send_file(os.path.join(REPORTS, request.args.get("name", ""))) # N3: send_from_directory rejects paths that leave the directory @app.get("/n3/report") def n3_report(): return send_from_directory(REPORTS, request.args.get("name", "")) # S6 CWE-918: server-side fetch of a request-supplied URL @app.get("/s6/preview") def s6_preview(): return requests.get(request.args.get("url", ""), timeout=5).text # N6: host allowlist checked before the fetch @app.get("/n6/preview") def n6_preview(): url = request.args.get("url", "") parsed = urlparse(url) if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS: return "forbidden", 403 return requests.get(url, timeout=5, allow_redirects=False).text ``` ## Reproduce the runs ```bash # The runs behind the results table (2026-10-10). Pin images by digest in CI. docker run --rm -v "$PWD:/src:ro" -v "$PWD/out:/out" -w /src semgrep/semgrep:1.180.0 \ semgrep scan --config p/default --metrics off --sarif-output /out/semgrep.sarif . bandit -r app -f sarif -o out/bandit.sarif # pip install 'bandit[sarif]==1.9.4' docker run --rm -v "$PWD:/repo:ro" ghcr.io/gitleaks/gitleaks:v8.30.1 dir /repo --redact docker run --rm -v "$PWD:/repo:ro" ghcr.io/gitleaks/gitleaks:v8.30.1 git /repo --redact docker run --rm -v "$PWD:/src:ro" ghcr.io/google/osv-scanner:v2.6.0 scan source -r /src docker run --rm -v "$PWD:/src:ro" aquasec/trivy:0.75.0 fs --scanners vuln,secret,misconfig /src docker run --rm -v "$PWD:/src:ro" -w /src ghcr.io/bridgecrewio/checkov:3.3.26 \ -d /src --framework terraform,dockerfile --compact --quiet # Score each SARIF file against the answer key. node score-sarif.mjs answer-key.json out/semgrep.sarif out/bandit.sarif ``` ## Decide what to adopt first Adopt in the order that buys the most risk reduction for the least tuning, and put each tool where its input appears: /domains/appsec/testing/devsecops-pipeline/ maps tools to stages, and the reference pipeline project on /labs/ shows them pinned in GitHub Actions and GitLab CI. | Situation | Start with | Then add | | --- | --- | --- | | Any team, any stack | Secrets scanning with history and push protection; SCA on lockfiles | SAST with diff-aware blocking on pull requests | | Web application with custom server code | SAST in pull requests plus a DAST baseline against each build | Authenticated DAST against staging; custom rules from your threat model | | Public or partner API with an OpenAPI or GraphQL schema | Schema-driven API testing with two test identities | SAST and rate-limit tests | | Containers and Kubernetes | Image scanning of the pushed digest; IaC scanning of manifests and Terraform | Admission control and runtime detection | | Java or .NET service with strong functional test coverage | SAST | IAST in the test environment, if your runtime is supported | | Many repositories and scanners, duplicate tickets | ASPM or DefectDojo for deduplication and ownership | SLAs per severity and owner | | Memory-unsafe code or complex parsers | Fuzzing with sanitizers | SAST rules for the parser's invariants | ## Pitfalls - Choosing on the vendor's demo application instead of your code. - Comparing tools on findings found without counting false positives and duplicates. - Reading a clean result as proof of security; a tool reports only what its rules and inputs let it see. - Enabling every rule at blocking severity on day one, which trains developers to bypass the gate. - Ignoring rule licensing: Semgrep CE is LGPL-2.1, but Semgrep-maintained registry rules carry their own license terms, and CodeQL is free only for open-source code, and for automated CI analysis only when that code is hosted on GitHub.com. - Buying ASPM before the scanners feeding it are tuned; aggregation of noise is still noise. ## Guided practice Your team is choosing a SAST tool for a Node.js and TypeScript codebase. Two trials on a seeded corpus produced the results below. Recommend one and say what else is needed. ### Supplied material ```text Seeded corpus: 24 flaws, 20 safe twins CWE-89 SQL built from strings 4 flaws / 4 twins CWE-79 HTML from user data 4 / 4 CWE-78 child_process.exec 3 / 3 CWE-22 path.join with user input 3 / 3 CWE-918 fetch of a user URL 3 / 2 CWE-1321 prototype pollution 3 / 2 CWE-639 missing ownership check 4 / 2 Tool X: found 17 flaws (all CWE-639 missed, 3 of CWE-1321 missed); flagged 6 twins 1,180 results on three real repositories; sample triage: 2.5 min per result, 22% real Tool Y: found 14 flaws (all CWE-639 missed, 4 others missed); flagged 1 twin 310 results on the same repositories; sample triage: 4 min per result, 61% real ``` ### 1. Compute the rates Calculate the true positive rate, false positive rate and Youden's index for each tool. Worked answer: X: TPR 17/24 = 71%, FPR 6/20 = 30%, index 41. Y: TPR 14/24 = 58%, FPR 1/20 = 5%, index 53. ### 2. Estimate running cost Estimate triage hours on the real repositories and the cost per real issue. Worked answer: X: 1,180 x 2.5 min is about 49 hours for about 260 real issues, roughly 11 minutes each. Y: 310 x 4 min is about 21 hours for about 189 real issues, roughly 7 minutes each. ### 3. Name the gap Which seeded class did neither tool find, and what covers it? Worked answer: CWE-639, missing ownership checks. No SAST tool knows the authorization rule; cover it with authorization tests using two identities, code review and an API test that tries cross-user access. ### 4. Recommend Pick a tool and say how you would close its specific misses. Worked answer: Y: a higher index and lower triage cost per real issue. Look at which four flaws it missed; if they cluster in one class, such as path traversal, add a custom rule with test cases and rerun the corpus before rollout. Deliverable: A decision record with per-CWE rates for both tools, the triage estimate, the chosen tool, custom rules to add and the classes covered by tests or review instead. Scope: The trial numbers are invented for the exercise. The seeded-repository results on this page were measured on 2026-10-10 with the versions listed and default configurations. ## Executable practice - [Repair a security pipeline for GitHub Actions and GitLab CI: project ZIP](https://softwaresecurity.com/downloads/projects/security-pipeline-365c6d54e88d6c1c.zip) — Node.js 22+ · no dependencies · GitHub Actions and GitLab CI YAML Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [OWASP Benchmark project](https://owasp.org/projects/benchmark) - [OWASP Benchmark for Java (v1.2 test cases and expected results)](https://github.com/OWASP-Benchmark/BenchmarkJava) - [OWASP Benchmark for Python](https://github.com/OWASP-Benchmark/BenchmarkPython) - [NIST SARD: Juliet C/C++ 1.3](https://samate.nist.gov/SARD/test-suites/112) - [NIST SARD: Juliet Java 1.3](https://samate.nist.gov/SARD/test-suites/111) - [NIST IR 8397: minimum standards for developer verification of software](https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8397.pdf). October 2021. - [NIST SP 800-53 Rev. 5 control catalog (SA-11, SI-7)](https://csrc.nist.gov/projects/cprt/catalog#/cprt/framework/version/SP_800_53_5_1_1/home). Release 5.1.1. - [OASIS SARIF 2.1.0](https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html) - [ZAP API scan](https://www.zaproxy.org/docs/docker/api-scan/) - [Schemathesis documentation](https://schemathesis.readthedocs.io/en/stable/) - [Semgrep licensing](https://docs.semgrep.dev/licensing) - [OWASP DefectDojo](https://owasp.org/projects/defectdojo)