If someone is selling you a network penetration test, and then running a vulnerability scanner and handing you a report, you’re not getting what you paid for, period.

Back around the turn of the century when I founded a small security consulting firm in North Texas, we had to explain the difference between these three types of engagements to most new customers and watch the color in their faces drain as they realized they hadn’t been getting what they were paying for from other firms.  Many times they just didn’t know how to ask for what they actually needed, so our first step in business development became customer education.  The differences between a network penetration test, an audit, and an assessment, can be summed up as follows:

A Penetration Test is generally exactly that. The target is attacked as a real attacker would; the approach is more stealthy, profiling is done to identify weak targets, those targets are enumerated and surgically attacked until one or more are successfully compromised, then the test is over. Penetration tests are generally limited in scope and comprehensiveness by nature, and on many occasions produce new, never-before known vulnerabilities in the systems and software the customer employs. Effort is focused on compromising the target in any way possible.  Penetration tests are generally the most expensive of the three types of tests I’m outlining because they by far require the most skill.  The differentiators here are the VARIABLE SCOPE and LENGTH of the test via selective targeting and the end of the test as soon as the goal is reached; successful compromise.

An Audit is generally a test for some form of compliance. The scope is defined exactly by whatever documentation outlines the requirements for compliance. This can be anything from multiple security compliance documents the size of HIPAA and SOX  to a simple checklist of specific vulnerabilities. If all the requirements are met, the target is compliant and has passed the audit. These are usually the next most expensive tests of the three; while they don’t require as skilled personnel as Penetration Tests, they’re more often than not extremely tedious.  The differentiation here is that you have requirements to audit the target AGAINST.

An Assessment is what you get from stock tools like vulnerability scanners, custom tools to identify vulnerabilities, and nice pretty reporting software to tie all the results together and provide references, mitigation and remediation guidance, executive summaries, a network health graph, threat metrics, and any number of other bells and whistles that can be compiled from the breadth of available information about the vulnerabilities found. The important differentiation here is that Assessments are essentially a test of the target for KNOWN vulnerabilities, hence the amount of available information to compile a pre-generated report from.  Any monkey can run packaged tools and generate reports from canned data, which results in Assessments generally being the least expensive of the three tests.

My firm was happy to provide all three (yes, we had various Nessus-flinging primates on call), given that the customer actually understood what they would be getting when they paid for one or the other.

—

An earlier version of this article originally appeared as a comment to a post over on Ari Takanen’s VoIP Security blog entitled “VoIP security auditing is becoming more and more complex … Not!”.  After posting the comment, I realized I had typed for quite a while and the content would actually make a decent blog post over here with the addition of a little more context information.