Software Securityhome
Fix a findingSearch
Phase 4 · Testing

Penetration testing

A penetration test is a time-boxed attempt by skilled testers to break into a system you choose, using the tools and techniques an attacker would, ending in a report of what they found and how to fix it.

Scanners and code review miss chained attacks and business-logic flaws, such as using one customer's order number to reach another's data. A test shows whether your own testing finds what an attacker would. The UK NCSC treats it as assurance for your vulnerability management, not your main way of finding bugs: ideally you already know most of what the testers report.

Who
An outside firm or an independent internal team, scoped with the system's owners and AppSec
When
Before launching new attack surface or a high-risk change, against an environment that matches production
How often
Before major launches, then on a regular cycle (often yearly) or as a contract or standard requires
You end up with
A report with each finding's risk, evidence and fix, what was not tested, and a retest of the fixes

Do this

  1. Decide what the test must answer

    Pick the system and the question: a new public API before launch, the boundary between tenants, the payment flow. Start from the threat model (Threat modeling), so the testers spend their days on the boundaries that matter most.

  2. Write a scoping document

    Agree it with the system's owners, engineers who know it and the testers. Record the technical boundaries, the types of test, the dates and effort in tester-days, any scenarios, what the testers need from you, compliance needs and how findings are rated, such as CVSS 4.0. This is the list the NCSC recommends.

  3. Give the testers accounts and context

    For software you build, prefer grey-box or white-box testing: accounts for every role and tenant, API documentation, and source code or design documents. NIST SP 800-115 notes that white-box techniques find defects in custom code more efficiently. Share your latest scan results too.

  4. Sign the rules of engagement

    Agree in writing the targets and addresses, the test windows, the accounts, the data that must not be touched, a stop procedure, emergency contacts on both sides, and immediate notice of any critical finding.

  5. Name a methodology and a baseline

    Ask which methodology the testers follow, such as the OWASP Web Security Testing Guide or NIST SP 800-115, and set a requirement baseline such as OWASP ASVS 5.0 level 2, so coverage and gaps are judged against something specific.

  6. Track findings as bugs, then retest

    File every finding in the normal backlog with an owner and a due date from your own risk rating, then have the testers repeat the original test cases against the fixed build. For each finding you did not already know about, work out which of your own controls should have caught it.

Done when

Each test has a written scope, and its findings are fixed or accepted with an owner.

Mark it in the gap check

Common mistakes

  • Reading a clean report as proof the system is secure. It covers only the scope and dates tested, and only issues known on the day of the test.
  • Testing your own application black-box to save setup time: the testers spend paid days finding out what documentation and accounts would have told them.
  • Choosing on price per day. A test's quality depends on the people doing it, so ask for redacted sample reports and the experience of the testers who will do the work.
  • Dropping routine security testing because a penetration test is booked. The NCSC warns that a planned test is no reason to stop your normal tests.
  • Leaving the retest out of the contract, so nobody confirms the fixes work.

Penetration test, bug bounty or VDP?

They answer different questions, so most teams add them in order: a vulnerability disclosure policy (VDP) first, penetration tests for depth, and a bug bounty once triage keeps up.

Penetration test, bug bounty or VDP?: table
QuestionPenetration testBug bountyVDP
Who testsA team you contractResearchers drawn by rewards, invited firstAnyone who finds something
CoverageDeep, within the agreed scope and datesBroad and continuous on in-scope assetsWhatever people happen to find
CostA fixed fee, usually priced in tester-daysRewards, platform or triage fees, staff timeStaff time to triage and fix
OutputA report with method, findings and limits, then a retestIndividual reports, paid by severityIndividual reports
Main riskA narrow test read as a clean bill of healthMore reports than you can triage or fixReports with nowhere to go

Tools and guides

Standards

NIST SSDF 1.1
PW.8 Test Executable Code to Identify Vulnerabilities and Verify Compliance with Security Requirements
OWASP SAMM 2
Security Testing

Sources

  1. UK NCSC: Penetration testing
  2. NIST SP 800-115: Technical Guide to Information Security Testing and Assessment. September 2008.
  3. OWASP Web Security Testing Guide. v4.2, the stable release; v5.0 in development.
  4. OWASP SAMM: Security Testing
  5. NIST SP 800-218: Secure Software Development Framework (SSDF) Version 1.1. February 2022.

Find your next answer

Type to search. Search runs in your browser.

Searches run in your browser, but the host can infer them from the index files they load. The address shows your search so you can bookmark or share it.