Avoiding metrics myopia

Things being measured are patently being observed in a very specific, focused manner. Things that are observed so closely tend to be presumed by others to be of concern to the measurer/observer, at least. Things under the measurement ruler therefore assume a certain significance simply by dint of their being closely observed, regardless of any other factors.

We see this 'fascination bias' being actively exploited by cynical promoters, marketers and advertisers on a daily basis through an endless stream of largely banal and unscientific online polls and infographics, their true purpose made all the more obvious by the use of bright primary-color eye-catching graphics. They are manipulating readers to believe that since something has been measured, it must be important. How many of us take the trouble to think about the quality of the metrics, or about all the other aspects that haven't been measured? Like bunnies in the headlights, we stare at the numbers.

In PRAGMATIC Security Metrics, we outlined 21 observer biases drawn from an even longer list drawn up by Kahneman, Slovic and Tversky in Judgment Under Uncertainty (1982): what I'm calling 'fascination bias' has some resemblance to what Kahneman et al. described as 'attentional bias', the tendency to neglect relevant data when making judgments of a correlation or association.

Fascination bias creates a genuine concern in that we tend to measure things that are relatively easy to measure, and place undue faith in those metrics relative to other factors that are not being measured. Back in 2011, Michel Zalewski said in his blog:

"Using metrics as long-term performance indicators is a very dangerous path: they do not really tell you how secure you are, because we have absolutely no clue how to compute that. Instead, by focusing on hundreds of trivial and often irrelevant data points, they take your eyes off the new and the unknown."

While we don't entirely accept that we 'have no clue how to compute security performance', his point about neglecting other risks and challenges due to being inordinately focused on specific metrics is sound.  It's only natural that what gets measured gets addressed (though not necessarily improved!). The unfortunate corollary is that what doesn't get measured gets neglected.

The upshot of this is that there is a subtle obligation on those who choose metrics to find ways to measure all the important matters, even if some of those metrics are expensive/complex/qualitative/whatever. It's simply not good enough to measure the easy stuff, such as the numbers that assorted security systems constantly pump out 'for free'. It's inappropriate to disregard harder-to-measure issues such as culture, ethics, awareness and trust, just as it is inappropriate to restrict your metrics to IT or cybersecurity rather than information security.

That's one of the key reasons why we favor the systematic top-down GQM approach: if you start by figuring out the Goals or objectives of security, expand on the obvious Questions that arise and only then pick out a bunch of potential Metrics to answer those questions, it's much harder to overlook important factors. As to figuring out the goals or objectives, conceptual frameworks for information security such as BMIS and ISO27k, based on fundamental principles, are an obvious way to kick-off the thinking process and frame the initial discussion.

ISO27k Toolkit

On the toolkit theme, I have just updated the FREE ISO27k Toolkit over at ISO27001security.com with an Excel workbook used to track progress on implementing the ISO/IEC 27001 and 27002 standards.

Thanks mostly to Ed Hodgson, the gap analysis/SoA workbook in the ISO27k Toolkit has been updated for the 2013 releases of the standards.

The new version has two main spreadsheets:

1. The first sheet is used to check and track progress towards implementing an ISMS complying with all the mandatory front parts of ISO/IEC 27001:2013 - mandatory, that is, if you intend to get your ISMS certified.  I made a few little wording changes and editorial decisions in this section, so if you use this for certification purposes, please double-check against the requirements formally specified in the standard and don't rely entirely on the spreadsheet!  The spreadsheet is not definitive.  The standard rules.
   
2. The second sheet covers the discretionary parts, namely the controls listed briefly in Annex A of '27001 and explained in more depth in ISO/IEC 27002:2013 plus any controls that you add or change on the list, for example additional legal, regulatory or contractual obligations, or ISO 22301, NIST SP800s or whatever.  Don't be afraid to adapt the list of controls!  '27001 Annex A and '27002 are intended to be 'reasonably comprehensive' starting points, laying out a decent set of good security practices, but your information security risks and hence control requirements are unique to you. 

For both parts, you simply select the relevant colour-coded status indicator from a drop-down list on each item, and record brief notes to explain the situation.  The status levels are adapted from Carnegie Mellon's Capability Maturity Model, showing progress from not-implemented-at-all (bright red) up to fully-implemented-working-and-auditable (dark green), plus grey options for "? unknown" (i.e. status not yet checked) and "Not applicable".

A third metrics spreadsheet simply counts the number of items at each status level in each of the two main sheets, and draws a pair of pretty pie charts showing the proportions ... 

These very simple metrics clearly indicate progress towards a compliant, working, provable ISMS managing a reasonably comprehensive suite of information security controls, as both pies gradually go dark green.  [Pies going green is usually a bad sign, but in this particular case dark green pies are tasty  :-)]   Actually, in Excel, it is easy to generate whatever format charts you like, line graphs showing the trends month-by-month for instance. That's left as an exercise for the reader.

As with all the ISO27k Toolkit items, it is provided free under a Creative Commons license that allows you to use and adapt it as much as you want for your own purposes, and to share it under the same terms (but not to sell it!). 

I take full responsibility for any errors in the spreadsheets.  If/when you find any errors, please let me know.  My Excel skills are rudimentary: I'm sure an Excel wizard would be able to come up with a sexier version, with better error-checking, better usability, more easily extendable, and perhaps more functional.  If you do improve it, please share it back for the benefit of the whole user community.

PRAGMATIC security metrics toolkit

Krag and I have been thinking about what might be of value in a 'toolkit' for security metrics.  The kinds of things we have in mind are:

• Resources such as books and standards on information security, risk management, governance, metrics, statistics and business management - the toolkit would contain references, reviews and links, not the actual content!
• Techniques, methods and approaches - naturally I'm thinking of the PRAGMATIC method but there are alternative approaches (such as GQM) that complement it: again, the toolkit would contain just a summary with pointers to further advice, since there is a lot to be said;
• Bootstrap metrics - perhaps a few suggested information security metrics to get you started?  I'm not so sure about this because it's hard to think of information security measurement requirements that are widely applicable, but I guess we could come up with a few illustrative metrics ideas.  Oh wait, we did that already - the 150 metrics in the book, most of which have been discussed on this very blog!
• FAQ - we have made a start on a security metrics FAQ but there's plenty more scope to develop that.  What questions would you like us to address?
• Do's and don'ts - lessons from the trenches.  We particularly favor the idea of publishing case studies on organizations that have developed coherent suites of information security metrics, discussing the process, pitfalls, triumphs and outcomes.  Would your organization be a guinea pig?  If so, get in touch.
• A project plan - perhaps some sort of generic outline or template of the steps most organizations would anticipate taking to specify, design, develop and implement an information security measurement system?  

So what do you think?  Is a "PRAGMATIC security metrics toolkit" something that would interest you?  What would you most like it to contain?  What else, apart from the items listed above, would be most useful for you?  

Go ahead, we're all ears.