Management awareness paper on email security metrics

Measuring the information security aspects of email and indeed other forms of person-to-person messaging implies first of all that you understand what your security arrangements are intended to achieve.  What does it mean to "secure email"?  If that's too hard to answer, turn it on its head: what might be the consequences of failing adequately to secure email? Does that help?

Our next metrics discussion paper opens with a brief analysis of the 'requirements and targets', also known as the objectives, of email security, expressed in broad terms. For instance, preventing or at least reducing the issues relating to or arising from spam and malware is a common objective ... hence one might want to measure spam and email-borne malware, among other aspects. 

That in turn begs questions about which specific parameters to measure and how - for instance, there are many possible ways to measure spam, such as the:

• Number of spam emails arriving at the organization, or rather the rate of arrival (spams per hour, day, month or whatever);
• Number of spam emails leaving the organization (!);
• Types of spams detected, perhaps analyzed according to the differing threats they represent, ranging perhaps from trivial/timewasting to targeted spear-phishing attacks;
• Proportion of spam that is detected and blocked versus that passed to users (and, hopefully, identified by them as spam);
• Cost of anti-spam controls, including the anti-spam software and systems, network and system load, user and support time spent on the problem etc.

For each of those potentially interesting concerns, there may be several actual ways to generate the corresponding measures, and several ways to analyze, present and hopefully use the data. The paper outlines just a few examples to illustrate the approach, but it should be obvious from the above that we could easily have written a very long and boring treatise just on email security metrics. We didn't ... because the paper was 'just' written for awareness purposes: it is designed to get managers thinking and talking about security metrics, opening their eyes to the possibilities and encouraging them to consider their own, specific information needs as opposed to the few generic examples given. Rather than boring the pants off our readers, we're hoping to intrigue and stimulate them, catch their imagination not write a book on it.  [By the way, that's one of the objectives for the security awareness program, implying that perhaps we might measure it in order to confirm whether the awareness program is achieving the objective, and if not suggest how we might do better!]

In case you missed it, there's an important point here with implications for all metrics (not just infosec metrics!). What we choose to measure depends on our information needs, organizational circumstances, objectives, challenges, maturity level and so on.  Your situation is different, hence you will probably be better served by a different set of metrics. Much as we would like to offer you a little pre-canned set of email security metrics, chances are they won't work terribly well for you. They'll be sub-optimal, perhaps costly, distracting and generally unhelpful. Remember this when anyone enthuses about particular security metrics, or when someone naively posses the classic "What metrics are best for X?"

One might argue that since we share a number of common information security challenges (such as spam), we might benefit from a number of common metrics ... but that's overly simplistic, just as it would be nuts to suggest that we should all employ identical anti-spam controls. Some email security metrics might well turn out to be more or less common than others but that alone is not a sensible reason to select or avoid them respectively.

This is why we invented the PRAGMATIC approach. While it is easy to come up with long lists of possible metrics using methods such as Goal-Question-Metric (see Lance Hayden's book "IT Security Metrics") and metrics catalogs (e.g. the 'consensus security metrics' from CIS), there was a distinct lack of guidance on how to shortlist and finally choose the few metrics actually worth implementing.