SMotW #16: policy noncompliance

Security Metric of the Week #16: number of security policy noncompliance infractions detected

The extent to which employees comply with the organization's security policies sounds like the kind of thing that management might want to track and, where appropriate, improve.  This week's metric is a typical, if rather naive attempt to measure policy compliance ... by counting noncompliance incidents.

Policies are 'mandated', in other words management expects everyone to comply with them unless there are justified reasons not to comply (meaning authorized exemptions for those organizations that are mature enough to appreciate the need to manage this aspect carefully).  While management originates most of the security requirements documented in policies, some derive from external obligations under applicable laws, regulations or agreements with third parties (e.g. PCI-DSS).  

The metric's wording implies that unauthorized noncompliance 'infractions' (more often called incidents) are being detected and recorded in a form that can be counted - typically some sort of security incident database, usually part of a Help Desk ticket management system.  Why not simply report the number of security incidents recorded by the Help Desk over, say, a month?  Such a metric would be low cost, but what about its benefits?

In reality, many other noncompliance situations occur, and some of them are detected or identified but for various reasons don't get reported as incidents.  As an example, who would bother reporting an everyday tailgating incident, even in a military organization that prides itself on physical security?  Furthermore, lots more noncompliance incidents are not even identified as such - they are not observed or recognized, or they are very short-lived or otherwise deemed trivial.  If an employee spots and challenges a tailgater, who then proceeds to identify themselves with an authentic-looking staff pass, the 'incident' is such a non-event that it is most unlikely to be reported, but it could of course be an actual intrusion.

All of this constitutes a huge bias to the metric as worded, a tremendous source of random error or noise in the measurement values.

Maybe it would help if we clarified the metric by reporting not the absolute number of policy noncompliance incidents but the rate of occurrence i.e. the number of incidents in a predefined period.  What would it mean if the metric jumped from 97 to 145 one month?  Is that something that should concern management?  What if it went from 145 to 97, or from 97 to 99, or hit zero?  How, exactly, would this metric support the decision making process?  Precisely what would management be expected to do with it?

Probing questions of this nature soon belie this metric's superficial allure.  It is not hard to find fault with it.  Arguably the most fundamental issue is that it is practically impossible to determine the true number of noncompliance incidents by direct observation, except perhaps in strictly controlled experimental conditions.  The best we can reasonably hope achieve in reality is to estimate the true number as rationally and accurately as we can, for instance by statistical means, using a more scientific process for identifying and reporting noncompliance incidents, such as periodic security policy compliance audits.  Unfortunately, that approach substantially drives up the Cost of the metric and so adversely affects its PRAGMATIC score:

P	R	A	G	M	A	T	I	C	Score
55	64	75	50	68	34	59	76	33	57%

We have been quite generous on the 75% rating for Actionability on the assumption that, if the measurements were poor, management would initiate whatever they considered appropriate to improve policy compliance, such as training and awareness activities coupled with increased management oversight, and perhaps more emphasis on enforcement actions.  We didn't have the same latitude with the rating for Accuracy, although using auditors or other professional assessors to measure the metric could improve its Independence, relative to self-reporting of noncompliance incidents by employees.

The Genuineness rating suffers largely because, if this metric were being reported and used proactively by management in an attempt to improve policy compliance, there is a distinct possibility that it would be deliberately manipulated.  There are some obvious if crude ways in which employees might 'game the system' to drive up the metric without materially improving compliance, such as consciously failing to report noncompliance incidents.  Even if management succeeded in addressing these tricks (e.g. by instituting and enforcing a policy on reporting incidents), other more subtle games would probably flourish.  It is amazing how creative people can get in the face of adversity!

The Predictability rating is also depressed because it is a backwards-looking metric: it tells us how things were in the preceding period but doesn't say much about how they may change in the forthcoming period, other than vague indications that might emerge from the background noise.


It is a reasonable assumption that security policies themselves are directly Relevant to information security, hence compliance with the policies is also Relevant.  However, there is more to security than policy compliance (it is 'necessary but not sufficient'), and as noted elsewhere the metric as worded does not perfectly reflect policy compliance, hence we rated the metric 64% on the Relevance criterion.  [This paragraph illustrates the PRAGMATIC thinking behind each of the ratings.  If we had the time and energy, we should probably document all the ratings on all the metrics for future reference, but in practice we would be more inclined to elaborate on and write-up the rationales for the few security metrics that we eventually adopt.]

The overall PRAGMATIC score of 57% tells us that, as originally stated, this is unlikely to feature as one of the few good security metrics we would chose, unless perhaps we were so short of inspiration that there were no higher-scoring candidate metrics on the table.  

Having discussed our concerns and hinted at some of the ways in which we might improve this metric, do you have some even better suggestions?  Or do you agree that this metric is essentially doomed?  Submit a comment and we'll do our best to respond positively.

Pre-PRAGMATIC

Hitherto - before the PRAGMATIC method was invented - deciding which security metrics to measure was a black art, a highly subjective decision making process.  One might even question whether organizations actually 'select' security metrics deliberately, systematically and rationally.  


Think about that for a moment.  Why does your organization measure whatever it does measure in relation to information security?  Does that mean that management doesn't care about all the other security stuff you could also measure?  Does it really matter what security metrics you use?


Pre-PRAGMATIC organizations presumably measure certain facets of information security because: 

• They are cheap and easy to report, typically because the raw numbers are readily available (some systems generate pretty graphs straight out of the box, but does management need them?);
• They are recommended by someone, peers claim to measure them, or the organization is required to report them by some third party (e.g. an authority such as a regulatory body or owner);
• 'It's the way we've always done it' ... (brains in neutral!);
• 'It seemed like a good idea at the time' ... although we may no longer recall why, and things have probably changed in the interim; 
• 'It's obvious!' ... but probe deeper and you may discover a mix of rational and irrational reasons for focusing attention on those particular aspects, and often a lack of appreciation of the opportunity cost i.e. it might be better to measure other things, or measure the same things in other ways;
• Management think they couldn't possibly measure the security things they really care about, and hence are forced to accept whatever metrics they are offered (nonsense!  Read Douglas Hubbard's book and get creative!).

Once an organization adopts the PRAGMATIC approach, selecting security metrics becomes a much more straightforward process.  The chosen metrics can be described and justified lucidly and convincingly, giving legitimate reasons for choosing them.  Other potential metrics can be assessed objectively in relation to the existing metrics suite, using the PRAGMATIC criteria as a framework for the decision-making process.  The PRAGMATIC approach lets management identify and weed-out low-scoring security metrics that aren't earning their keep, not just saving the associated metrics generation and reporting costs but focusing finite management attention on more valuable metrics.

SMotW #15: HR security maturity

Security Metric of the Week #15: Human Resources security maturity

In order to explain the PRAGMATIC score for this week's example security metric, we first need to introduce you to the concept of security maturity metrics.  Bear with us.


Section 8 of ISO/IEC 27002:2005 lays out a suite of HR-based information security controls that apply to the pre-, para- and post-employment phases.  For example, prior to offering anyone a role, especially in a powerful/trusted position, wise organizations conduct suitable background checks to weed-out unsuitable candidates.  For highly sensitive government and military work, the security clearance process can involve an extensive range of checks including credit worthiness, criminal history, identity, qualifications, professional experience and character references, in addition to a structured interview process and on-the-job supervision/oversight during a probationary period.  For low-grade positions such as office cleaners, pre-employment checks tend to be trivial and in many cases are left to third parties supplying contract staff ... but while this is common practice, it creates information security risks that probably deserve more attention.  Office cleaners typically work alone out-of-hours and have ready access to IT equipment, paperwork and other valuables:   do you really want to let someone of unknown vintage and negligible loyalty, paid a minimal wage, loose in your office?

So-called maturity metrics are an excellent way to measure such complex situations.  The idea is simply to lay out a spectrum of good practice security controls ranging from trivial, negligible and weak up to extensive, leading-edge and strong.  Figuring out where an organization stands in relation to the spectrum of controls allows us to determine a maturity level or score. 

Maturity metrics are similar to the Capability Maturity Models originally created [and trademarked] by Carnegie Mellon University as a means to assess and measure software development practices.

Given the number of aspects relevant to information security, over the course of about two decades we have developed a tabular style maturity metric with rows for each type or area of control and columns for key points on the scale.  Cells in the body of the table contain examples of the controls, providing sufficient information to guide a competent reviewer in determining the most appropriate level and hence the score.

While scoring process is inevitably subjective, stating the scoring criteria makes it objective enough that different reviewers, auditors, managers, assessors etc. can gather, consider and discuss the evidence, generally reaching agreement on specific maturity scores.  


We have provided a suite of security maturity metrics based on the recommendations in ISO/IEC 27001 and 27002 as an appendix to our book.  As an example, here is one of the seven rows in the table covering section 8's HR security controls:

No human resources security	Basic human resources security	Good human resources security	Excellent human resources security
Information security rôles and responsibilities are entirely undocumented	Some information security rôles and responsibilities are documented, though not very well or consistently	Most information security rôles and responsibilities, including all the important ones, are assigned to individuals through being incorporated into vacancy notices, job descriptions and/or codes of conduct	Information security rôles and responsibilities are comprehensively documented, formally assigned to suitable individuals (typically in legally-binding contracts of employment or terms and conditions of employment), and are  proactively maintained (e.g. periodically reconfirmed with the individual’s signature to confirm their acceptance)

The four columns correspond to maturity scores of 0%, 33%, 67% and 100% respectively.  There is a further implied scoring point at 50%, marking the divide between practices that are generally considered unacceptable and those that are broadly acceptable.


OK, so now let's look at the PRAGMATIC rating for this kind of metric:

P	R	A	G	M	A	T	I	C	Score
90	95	70	80	90	85	90	85	90	86%

The metric scores remarkably well, especially given that, while HR practices are undoubtedly relevant to information security, they are rather difficult to measure.  In fact, to our knowledge, very few organizations have decent security metrics in this area.  Most seem content with classical HR metrics such as the number of employees that have completed some form of security training, despite the fact that such metrics do not adequately reflect security in practice.  Did anyone actually learn anything from the training?  Did they actually change their behaviors as a consequence?  Or did they just turn up under sufferance and passively sit there just to get the tick on their personnel records?  The classical metric says practically nothing about these aspects.  [Feel free to determine the PRAGMATIC score for the classic metric as a homework exercise.  We'd be amazed if it scores anything remotely approaching 86%!]


There are further security maturity metrics in the book, corresponding to the remaining sections of ISO/IEC 27001 and 27002.  We'll expand on them in future metrics of the week but for now you will have to bide your time until the book is published, although we will try to address any questions in the comments to this blog.

PRAGMATIC Security Metric of the Quarter #1

PRAGMATIC Security Metric of the First Quarter

Having scored and discussed fourteen Security Metrics of the Week during the first three months of this blog, it seems appropriate now to take a step back, review the metrics we have discussed thus far and consider the value of the PRAGMATIC process.  

Here are the fourteen security metrics, tabulated in descending order of their overall PRAGMATIC scores.  Click any of the metrics to read more about them and discover why we scored them thus.

Metric	P	R	A	G	M	A	T	I	C	Score
Discrepancies between physical location and logical access location	75	76	72	90	82	75	85	83	60	78%
Information security policy coverage	75	82	92	78	80	70	73	60	81	77%
Unowned information asset days	40	51	84	77	74	86	92	94	82	76%
Number of unsecured access points	95	80	90	70	85	77	45	75	55	75%
% of critical controls consistent with controls policy	83	92	80	83	89	82	32	70	35	72%
Days since logical access control matrices for application systems were last reviewed	55	80	95	30	80	85	60	70	80	71%
Number of unpatched technical vulnerabilities	80	64	80	70	80	75	25	85	52	68%
Coupling index	68	85	50	60	72	47	35	61	42	58%
Vulnerability index	74	85	71	74	60	32	46	33	19	55%
System accounts-to-employees ratio	74	67	38	39	68	42	36	83	44	55%
Corporate security culture	80	80	60	55	75	55	10	45	10	52%
% of purchased software that is unauthorized	71	51	90	75	82	35	13	20	6	49%
Security budget as % of IT budget or turnover	13	3	16	2	2	0	4	18	88	16%
Number of firewall rules changed	2	1	1	10	2	33	14	4	17	9%

In simple numerical terms, the metric Discrepancies between physical location and logical access location is the leader of this little pack which qualifies it as <cue drum roll> our first PRAGMATIC Security Metric of the Quarter.  In fact there's clearly not much to choose between the top four metrics in the table in terms of their overall PRAGMATIC scores.  The scores and hence rankings may well have changed if we had made different assumptions in the scoring, or of course if we had altered the specification/wording of individual metrics to address the issues we identified and hence altered their scores.  Furthermore, your scoring of the metrics may differ from ours due to  differences in how we each understand and interpret both the metrics and the PRAGMATIC approach.  We don't have the same experience as you, our biases differ, our presumptions and organizational contexts differ and no doubt we have different potential audiences and purposes in mind for these metrics.

That whole line of discussion is moot, however, since we are not claiming that the PRAGMATIC approach is scientific and objective.  Our top-scoring metric is not necessarily the best of the bunch under all circumstances for all organizations, just as the lowest scoring metric may be appropriate and score more highly in certain situations.  The approach simply offers a rational way to consider the value of and compare various security metrics, to elaborate on their pros and cons, to identify ways in which the metrics might be re-phrased or materially altered to improve them, and most of all to facilitate a more informed and productive metrics discussion with management.  Even if you simply use the process to shortlist the most promising from a bunch of metrics candidates, leaving the final selection to management, isn't that a worthwhile outcome?

There's plenty more to say yet about being PRAGMATIC, including ways to glean further useful information from the data in the scoring table above, but we'll leave that for the book, future blog pieces, seminars and papers.   Meanwhile, do please let us know about your favorite security metric and we'll see we make of it.  We look forward to your comments on this blog and emails, especially constructive criticism and creative ideas to make the PRAGMATIC approach even more effective.  Over.

SMotW #14: logical access reviews

Security Metric of the Week #14: days since logical access control matrices for application systems were last reviewed

The fundamental premise for this metric is that if the logical access controls within [shared] application systems are reviewed more often, they are more likely to reflect the corresponding business requirements (for example, taking account of the differing roles that people play, the comings-and-goings from the workforce, changes in business processes and IT systems over time, and compliance with the access rules defined by information asset owners).  In other words application systems are more secure than if the access reviews are infrequent or nonexistent. 


In relation to Acme Inc., we thought this metric might be used and reported 'per application system', measuring how effectively access rights are being managed for each system.  Let's discuss the PRAGMATIC numbers in this context:

P	R	A	G	M	A	T	I	C	Score
55	80	95	30	80	85	60	70	80	71%

The metric scores strongly in terms of being Actionable: clearly, if the access rights have not been checked for a long time, they ought to be checked soon.  But how long is 'a long time'?  It is implied that review periods are for various classes or types of systems, and measuring discrepancies between actual and target dates in days: some matrices will have been reviewed on or before their targets, others will be overdue.  We presume that the raw data - target and actual dates - may be read from some sort of inventory of the systems, populated in turn from the access matrix review and update procedures.  This metric also scores highly in terms of Accuracy, given that the number of days between target and actual dates are readily measured and checked, although there is still a practical question of determining the date an access matrix is declared ‘fully reviewed’.  It is quite conceivable that this date might be deliberately manipulated by someone seeking to improve their score (for example conducting sham reviews), so the metric is rated down on Genuineness, although the metric should be based on verifiable facts (namely dates taken from evidence of the reviews), hence its Independence or Integrity score is not too bad.


On Predictability, the metric rated just over 50% because although application access controls are quite important for information security, they are clearly  insufficient to constitute a complete or comprehensive security approach.  Many other forms of control are needed to secure an organization's information.  However we were quite generous on the score because, in our experience, organizations that are sufficiently on top of their application security to review the access rights regularly tend to be generally competent and mature at information security - in other words, a high score on this specific metric is, we feel, an indicator of a reasonable level of information security.

The metric itself is relatively straightforward and simple to understand, hence it scores well for  Meaning.  It didn't quite merit 100% because, as presently worded, it begs questions ranging from 'What are logical access control matrices?' and 'What are application systems?' to 'How many systems are there?', 'Who does the reviewing?' and 'How do they review - what is a review anyway?'   If the metric was clarified on issues such as these, it should be possible to drive its score up to the point that it becomes a serious contender as a corporate security metric, but its ultimate fate depends on whether there are other, better security metrics for the security aspects that matter most to the organization.


Whereas we have been thinking about using the metric in the context of individual application systems, it could also be used to measure, report and track days-since-last-reviewed across the entire portfolio of applications throughout the organization.  Doing so would highlight those application systems whose access rights have never been reviewed since the day they were implemented - a surprisingly common occurrence in practice.  Provided there are suitable controls in place to prevent people ignoring or discounting specific application systems simply in order to improve the metric, it will in effect pressure management into conducting reviews on the worst-scoring application systems.  The days-since-last-reviewed numbers from application systems might even be weighted to reflect the relative importance of access controls to those systems, putting yet more emphasis on the applications most in need of review, but the additional complexity, subjectivity and Costs of such an approach would not help its PRAGMATIC score.

Once again this week, we've demonstrated how the PRAGMATIC approach leads us to delve below the superficial appearance of a candidate security metric, consider its strengths and weaknesses, and identify specific areas where the metric might be improved.  We hope you are starting to see the power of being PRAGMATIC.