SMotW #31: security metametrics

Security Metric of the Week #31: security metametrics measuring the quality of security metrics in use

While we were writing the book, Krag and I often bantered about the true meaning of the word 'quality'.  We may not have achieved true consensus, but we enjoyed our often tongue-in-cheek discussions and we do at least appreciate the point that different people read different things into it.

With that in mind, you may well interpret this week's candidate metric somewhat differently to us, and that's fine.  Furthermore, given that we patently have a passion for metametrics (information about metrics), we freely admit to having a distinct bias towards any measure of the quality of security metrics, so in a sense this is a useful test of the PRAGMATIC method.  Let's see whether we can convince you that security metametrics or indeed the PRAGMATIC approach deserves its outstanding PRAGMATIC score shown below.

We firmly believe the quality (whether that means inherent quality, suitability or fitness-for-purpose) of the organization’s information security metrics has an absolutely huge bearing on the way management governs, directs, controls and manages its information security, and hence on the security outcomes achieved.  We may be charged with reductio ad absurdum but, for the sake of discussion, contrast these dramatically different scenarios:

1) On the basis that 'metrics' qualifies as perhaps the most advanced, complex and difficult facet of information security management, an organization that had utterly superb security metrics would undoubtedly be classed as highly mature in this sphere.  It would have a comprehensive suite of security metrics in place, having consciously elaborated its security measurement requirements, designed the measurement system and selected appropriate metrics to address its strategic, business, governance, compliance, risk and security management goals.  The measurement system would be running sweetly, being an integral part of the  Information Security Management System which would likewise be of high quality.  Management would truly appreciate their wonderful security metrics, using them routinely and systematically for decision making (e.g. allocating scarce resources to security activities that add or protect most value) and finding that, on the whole, the metrics earn their keep by consistently providing useful, reliable information in areas that would otherwise be extremely tricky to measure, direct and control.  Key security decisions would be driven by factual data, and managers would generally trust both the raw numbers and the analysis and presentation.  The rare occasions in which their professional advisers or gut instincts indicate a course of action different to that suggested by the metrics would themselves be interesting points for discussion, presenting opportunities for deeper analysis and understanding and, where appropriate, improvements to the metrics system.  The assurance that stems from management's supreme confidence both in the metrics and hence in their knowledge of the organization's security status would be almost palpable, and that in turn would reassure various stakeholders that security was in fact being extremely well managed.  Inevitably some security incidents would still occur, but they would be relatively minor and efficiently handled.  The organization would be spared its worst security nightmares having invested appropriately in the areas of security that mattered the most.

2) Assuming they were even aware of their existence, managers in an organization with totally crappy security metrics, on the other hand, would openly distrust and discount whatever security information actually reached them.  They would be deeply suspicious or cynical about practically everything they read on security.  They might still smart from recent situations in which the numbers clearly lied to or misled them, failing to pre-warn them about impending security issues that led to shocking incidents, unanticipated corporate losses and personal embarrassment.  Entire classes of information security controls and risks would be completely unmeasured, including crucial aspects such as security awareness that would [wrongly] be deemed 'inherently unmeasurable'.  Inadequate resources would be allocated quite randomly to security, bearing little relation to the associated security risks which would be poorly understood and inappropriately ranked, both comparatively and in relation to other business risks.  Significant security improvement opportunities would remain unrecognized and neglected, since there would be negligible feedback and perhaps grossly misleading information on the actual security status.  Important security decisions would be driven mostly by guesswork and assumptions, with no rational basis in fact.  Specious arguments such as "It's the way we've always done it" or "We need only do the minimum to achieve compliance" could not easily be challenged due to the abject lack of information to the contrary.

The PRAGMATIC approach is of course a great way of assessing and scoring the quality of the organization's security metrics, and hence qualifies as a valuable metric in its own right.  It can be used to compare candidate metrics on a rational basis, identify opportunities for even better security metrics and drive through improvements by generating Predictive, Relevant, Actionable, Genuine, Meaningful, Accurate, Timely, Independent and Cost-effective information about the metrics.

In the imaginary context of Acme Inc, we scored the metric thus:

P	R	A	G	M	A	T	I	C	Score
96	91	99	92	88	94	89	79	95	91%

Please note that, although this is our pride and joy, we could not honestly justify straight 100% ratings across the board!  In particular, we acknowledge that our obvious passion for the subject means we are not entirely Independent in rating the metric we created, and there are always opportunities for improvement.  Nevertheless, we commend the PRAGMATIC approach to the house.