Penetration testing is one of the least transparent things a business buys. The deliverable is a document, the work is invisible, and unless you already have the expertise to evaluate it, a thorough manual assessment and a lightly edited vulnerability scan look remarkably similar from the outside.
These are the questions we would ask if we were the buyer rather than the supplier.
On what the test actually is
- How many hours of manual testing are included, as distinct from automated scanning? A credible answer is a specific number. Vagueness here is the single most reliable warning sign.
- What proportion of findings do you expect to come from manual work? If the honest answer is very few, you are commissioning a scan, which is a legitimate product but should be priced as one.
- Which methodology do you follow? Expect a specific reference such as PTES, the OWASP Testing Guide or OSSTMM, and expect them to be able to describe how it shapes the work.
- Will you attempt to chain findings together? Individually low-risk issues frequently combine into a critical path. Chaining is where manual testing earns its cost.
On the people doing the work
- Who specifically will test our environment, and what are their certifications? Named testers with credentials such as OSCP, CREST or equivalent, not an anonymous pool.
- Will the person who did the testing be on the debrief call? If the tester cannot be questioned directly by your engineers, ask why.
- Do you subcontract? Not automatically a problem, but you should know, and it belongs in the contract.
On the deliverable
- Can we see a redacted sample report before we commit? A serious firm will have one ready. Look for reproduction steps precise enough for your developers to follow, evidence for each finding, and severity ratings that reflect your business context rather than raw scanner scores.
- How are severities determined? A default CVSS score with no adjustment for exploitability in your environment tells you the ratings have not been thought about.
- Is a retest included after we remediate? It should be. A finding is not closed until someone has confirmed the fix works.
On scope and safety
- What exactly is in scope, and what is excluded? Ambiguity here produces the most common post-engagement dispute. Get the target list, the IP ranges, the application roles and the excluded techniques in writing.
- What is the communication protocol if you find something critical mid-test? You want immediate notification for a live compromise or a critical exposure, not a surprise in the final report three weeks later.
A note on price
The cheapest quote is usually the cheapest because it contains the least human time. That is not automatically wrong; if you genuinely need a compliance tick against a small, well-understood scope, a lighter engagement may be entirely appropriate and you should not overspend.
But if you are buying a test because you want to know whether a competent attacker could get in, hours of skilled human effort are the product. Anything that appears to deliver that outcome without those hours is worth a second look.
One last suggestion: ask each vendor what they would not test and why. The answers are revealing. A firm that is honest about the limits of the engagement is usually being honest about the rest of it too.
Written by the Techx4u, Inc engineering team. If you would like any of this looked at in your own environment, get in touch.