Security metric #55: policy coverage

Information Security Metric of the Week #55: information security policy coverage of frameworks such as ISO/IEC 27000

In much the same way that two-dimensional maps of three-dimensional landscapes are useful for hill-walkers, various frameworks, standards and methods such as ISO27k, SP800-53, COBIT and the Standard of Good Practice are useful guides for navigating the field of information security.  Just as cartographers must transform the literal land into graphic representations on the map, the standards bodies and assorted authors take somewhat arbitrary decisions about which elements of information security to cover and in what sequence.

For example, section 7 of ISO/IEC 27002:2005 covers two distinct but related issues in asset management: 7.1 Responsibilities for protecting information assets, and 7.2 Classification of information.  Those two aspects could have been scoped and titled differently and might have been placed in separate sections or incorporated into other sections of the standard but the ISO/IEC committee, in its wisdom, chose to cover them both together in section 7.  

Security responsibilities and information classification are relevant to various information security risks and control objectives, hence they (along with most other controls) could have been discussed from different perspectives in several parts of the standard.  However this would have created duplication and confusion.  Instead, the controls are each discussed once and, where necessary, cross-referenced elsewhere.

ISO/IEC 27002 provides a convenient map that is widely understood.  Aside from the structure - more importantly in fact - the standard lays out a reasonably comprehensive suite of information security controls that could be considered a basic or minimal set: with some exceptions, most organizations that take information security seriously are using most of the controls listed in the standards.  Therefore, comparing an organization's information security controls against those recommended in the standard to identify any gaps is one way to measure the comprehensiveness of its controls.

That said, ISO27k is imperfect.  Aside from issues with the wording and meaning of the standard when it was published, there is a further dynamic aspect.  ISO/IEC 27002:2005 has become outdated in various respects, for example it does not explicitly and comprehensively cover cloud computing since cloud computing was barely even conceived when the standard was drafted.  With some artistic license, several recommended controls in the standard can be interpreted in the cloud computing context, but other necessary controls are either completely missing from the standard or are of limited value as currently worded.  To fill-in the gaps, we could wait for the standard to be updated and released (later this year, hopefully), or we could use various other security standards and frameworks in the meantime, supplementing them with advice from information security, risk, compliance, governance and related professionals, tailored to our specific circumstances.

Against that background, let's look at the value of a metric that measures the extent to which the organization's security policies cover the entire security landscape.

When they assessed this metric using the PRAGMATIC method, ACME management had in mind using their own information security coverage map which had been drawn up by the CISO to reflect the common ground across several security standards.  They envisaged the CISO systematically checking for discrepancies between the suite of policies and ACME's map and drawing up a simple color-coded coverage diagram similar to that shown above - red meaning "Inadequately covered", amber being "Partially covered" and green for "Fully covered".

P	R	A	G	M	A	T	I	C	Score
70	75	90	69	85	76	72	65	85	76%

  

The managers recognized a potential bias in that the person assessing and measuring the policies also owns them.  The CISO might honestly believe that one or more given ACME policies entirely cover part of the coverage map, whereas another security professional might feel that the policies don't go far enough to address the associated risks.  They could get around this limitation by commissioning an independent consultant or auditor to assess and measure the policies, and perhaps by separately measuring the correspondence between their information security map and applicable standards.  They might even go as far as to adopt the excellent Unified Compliance Framework, a rigorous synthesis of information security-related recommendations and obligations drawn from practically all the standards and laws in this area.  On the other hand, all that extra work would markedly delay the production of the metric and increase the costs.  A more pragmatic approach might be to have someone from Internal Audit or Risk Management cast a cynical eye over the scoring and challenge the CISO to justify her decisions - a process known as normalization in the world of metrics.  The CISO would also be asked to make notes during the measurement which would be useful for planning updates both to the policies and to the coverage map ... and here we're already talking about using the metric to inform decisions, implying that it definitely has potential.  In summary, the metric's 76% PRAGMATIC score feels right.

This is just one of a few similar metrics discussed in the book, and it would not be hard to think up many more along these lines, including variants of the ones we have discussed, similar metrics proposed elsewhere, and novel metrics invented for this purpose.  The PRAGMATIC method enables us to analyze and compare the metrics in a rational and systematic way, forcing us to think through the pros and cons of each one before selecting "a few good information security metrics".  We don't mean to trivialize the effort required to complete the metrics design, specify any mathematical analysis and presentation, implement them and of course use them, but PRAGMATIC gets us over by far the biggest obstacle: selecting the right metrics.