Security auditing: what the words actually mean

A director asks for a security audit. Three suppliers quote, and the quotes differ by a factor of ten. They are not competing on price — they are quoting for different work, because the word means something different to each of them.

The business picks one, receives a document, and discovers it does not answer the question that prompted the request. The money is spent and the question is still open.

Getting value here depends almost entirely on knowing which of four distinct things you are buying, and that is decided before any supplier is contacted.

The four things people mean

NIST's technical guide to information security testing, SP 800-115, is useful for the underlying distinction: examining a system and testing it are different activities with different benefits and limitations. Examination inspects configuration, policy and documentation. Testing exercises the system to see how it actually behaves.

In commercial practice that splits into four offerings.

The blind spot column is the one worth reading twice, because each of these is regularly bought as though it answered all four questions.

Vulnerability scan

Automated, cheap, repeatable, and the right thing to run continuously rather than annually. It compares what it can see against a database of known issues.

Its limitation is that it reports what is theoretically vulnerable without knowing what is reachable or what matters here. A scan finding two hundred issues has not told you where to start, and a report of two hundred issues with no prioritisation is a common reason nothing gets fixed.

Penetration test

A person attempting to achieve defined objectives against defined targets. Valuable because it establishes what is actually exploitable in your specific configuration rather than what is theoretically vulnerable.

Its limitation is scope and time. A test covers what it was pointed at, within the days purchased. A clean result means nothing was found in that scope in that time, which is a genuine but bounded assurance — and it is routinely reported internally as though it meant the business is secure.

Security assessment

A structured review of controls against a framework, usually involving interviews, document review and configuration inspection. This is the one most growing businesses actually need, and the one least often bought, because it sounds less definitive than a test.

It answers the question that usually prompted the request — are we doing the right things — which neither a scan nor a test addresses.

Compliance audit

Verification against a specific standard, usually because a customer, regulator or insurer requires it. Necessary when it is required and close to useless when it is not, because the standard's scope decides the audit's scope regardless of where your actual risk sits.

Which to buy, and in what order

For a business that has not done this before, the order matters more than the choice, and the intuitive order is backwards.

1. 1

   Start with an assessment against a published control set

   CIS Controls Implementation Group 1 is a reasonable baseline. This tells you which controls are absent, which is the information most likely to change what you do next.

2. 2

   Implement what the assessment found

   The findings at this stage are typically inventory, account management, patching, backup and logging. None of them requires a test to discover.

3. 3

   Introduce continuous vulnerability scanning

   Scanning is a process rather than an event. Monthly at minimum, with the results going to someone who owns remediation.

4. 4

   Then commission a penetration test

   Once the basics are in place, a test finds the things only a test can find. Before that, it finds the basics, expensively.

5. 5

   Add compliance audits when someone requires one

   And treat the requirement as the reason, rather than as evidence that this is the right next security investment.

The most common error is starting at step four. A penetration test against an estate with no asset inventory and no patching process produces a report enumerating the absence of an asset inventory and a patching process, at the highest day rate in this article.

Prioritising what comes back

Every engagement produces findings and most findings never get fixed, because the list is long, everything is labelled critical or high, and there is no basis for choosing.

Severity scores are the usual basis and they are insufficient on their own. CVSS describes the characteristics of a vulnerability in the abstract. It does not know whether the affected system is internet-facing, whether it holds anything sensitive, or whether anyone is actually exploiting the issue in the wild.

That last input is the most useful and the most overlooked. CISA maintains a Known Exploited Vulnerabilities catalogue — a list of vulnerabilities with confirmed active exploitation. A vulnerability on that list is being used against real organisations now. A high-severity vulnerability that appears nowhere in it may never be used at all.

Two inputs the score cannot supply are doing the work here: whether it is being exploited, and whether it is reachable. Both are cheap to establish and both change the order substantially.

Getting a useful engagement rather than a document

The commercial side is where most of the value is won or lost, and it is decided before the work starts.

• Write the question you want answered, in one sentence, before contacting anyone. If suppliers propose different work, the question was ambiguous.
• Define scope explicitly, including what is excluded and why. Undefined scope is how a test misses the thing you were worried about.
• Ask what the deliverable looks like, and ask to see a redacted example. A tool output with a cover page is a common deliverable and is not worth a consultancy day rate.
• Require findings to be prioritised for your environment, not only scored. Otherwise prioritisation lands back on you, which is the part you were paying to have done.
• Agree a retest, and agree what it costs. Findings closed without verification are findings believed to be closed.
• Ask who does the work, not who sells it. On a test the individual matters more than the firm.

One more, which sounds obvious and is regularly skipped: decide before the engagement who will own the remediation and where the time will come from. A report with no owner produces a second report next year describing the same findings.