SMotW #85: controls consistency

Security Metric of the Week #85: consistency of information security controls

This metric implies that someone is concerned about security controls being inconsistent, but what does that mean - inconsistent in what regard? Possible types of inconsistencies include:

• Controls do not sufficiently mitigate the risks, address the wrong risks, or are in some way inappropriately designed/specified;
• Expected or standardized controls (e.g. controls mandated in law) not implemented in all relevant places;
• Controls not implemented to the same degree or extent, or in the same way, in all relevant places;
• Controls that vary over time (e.g. security procedures ignored in busy periods);
• Controls not operated or managed in the same way in all relevant places;
• Others.

ACME's senior managers did not rate this metric highly, being concerned about its Accuracy, Timeliness, Independence/integrity and Cost-effectiveness:

P	R	A	G	M	A	T	I	C	Score
78	83	67	60	71	33	27	31	27	53%

However, from the perspectives of the CISO or ISM, the metric was more PRAGMATIC:

P	R	A	G	M	A	T	I	C	Score
85	90	76	60	90	50	46	100	75	75%

They could see themselves using this metric to drive up consistency of security controls in whatever respects they chose to measure ... although exactly how they would measure consistency was not exactly self-evident: they were thinking initially about using and perhaps extending their routine compliance checks against ACME's baseline security standards.

Notice the distinctly different ratings for Independence/integrity given in these two PRAGMATIC assessments. In the former, senior management were concerned that if they started using the metric to pressure Information Security and various business units to improve their information security, things might deteriorate to arguments over the measurements rather than productive discussion around making necessary improvements. They also weren't entirely convinced that the metric would be a trustworthy guide to controls consistency. In contrast, the CISO and ISM envisaged measuring the metric themselves for their own purposes in connection with continuously improving ACME's ISO27k Information Security Management System, with little need for discussion or argument with those being measured. In fact, the metric might not even need to be reported or circulated beyond the infosec office.

This is a good illustration of why published lists of security metrics (including the 150 examples in our book!) are of dubious value except perhaps as creative inspiration. Despite what you might think, a security metric that works brilliantly for one organization may be mediocre or quite inappropriate for another, while one that is ideal for a particular purpose and a specific audience within a given organization may be a poor choice in other circumstances or for other audiences. This is precisely what makes the PRAGMATIC method shine: it offers a systematic, structured way to figure out and compare the merits of various possible security measures in a specific situation or context, something that was previously very difficult to achieve.

With that, we'd like to wish all our readers a brilliant Christmas: the next SMotW will appear here early in the new year, although we might perhaps blog about new year's metrics resolutions. Meanwhile, we hope Santa brings you all you desire, and doesn't get stuck in the chimney.

Merry Christmas from Gary & Krag. Have a good one.