Metric of the Week #50: policy clarity

Information Security Metric of the Week #50: proportion of information security policies that are clear

This week's worked example concerns a metric that looks, at first glance, to be far too subjective to be of any value ... but read on.

Clarity is a critical requirement for information security and indeed other kinds of policies.  Policies that are garbled, convoluted, rambling and full of jargon are less likely to be read and understood by the people that ought to be compliant.  As a corollary, well-written, succinct policies facilitate reading and understanding, while also making them 'motivational' in style encourages compliance.

It's obvious, isn't it?  So how come we still occasionally see policies written in the worst kind of legalese, stuffed with obscure/archaic language in an embarrassingly amateurish attempt, presumably, to appear "official"?

The suggestion for this metric involves regularly surveying employees' opinions regarding the clarity of the security policies, using a questionnaire based on a Likert scale developed and administered in person by a market research firm.   

Applying the PRAGMATIC method, ACME managers expressed some interest in this metric, but were concerned on a number of fronts:

P	R	A	G	M	A	T	I	C	Score
75	70	68	41	96	50	56	90	34	64%

There are at least three advantages to this metric, the first one more obvious than the others:

1. Aside from the overall clarity/readability level, the metric should identify policies that are perceived as better or worse than average, providing shining examples and opportunities for improvement, respectively.  This is useful new information for the authors of policies, making this a beneficial operational security metric. 
2. Asking employees about the policies will inevitably find some unable or unwilling to state an opinion because they cannot recall reading the policies.  The proportion of 'no response' returns is therefore a rough measure - an indicator - of the extent to which policies are actually being read.
3. Asking employees about policies also prompts them, and hopefully their colleagues, to (re)read the policies in order to give an opinion ... which is itself a useful awareness outcome.

On the downside, however, the metric would be Costly and slow to collect if a market research company was engaged.  It could be run more cheaply by employees such as Information Security, although since they are generally responsible for the policies, they may be tempted to influence or manipulate the metric to make them appear clearer than they really are - or at least they would find it difficult to prove that they were completely unbiased.  Increasing the Cost-effectiveness rating in this way would therefore depress the Independence factor and, perhaps, the Genuinness and Accuracy factors which are already low due to this being such a subjective matter.

There is a further advantage to having members of Information Security conduct the surveys in person: it would give them a legitimate reason to leave the sanctuary of the security office, get out among their colleagues and pick up on things that are really going on in the business.  Their colleagues, at the same time, would get to meet and chat with Information Security people in an informal, non-confrontational situation, and could perhaps raise other queries or concerns.  This would be particularly valuable if the security team was reclusive and shy, or appeared aloof and distant.

[A variant of this metric involving an automated assessment of clarity was also tabled for consideration by ACME management: it will appear on this blog at some point so you can either wait patiently to find out about it or look it up in Chapter 7 of the book!]

Metric of the week #49: infosec risk score

Security Metric of the Week #49: information security risk score

Most risk analysis/risk assessment (RA) frameworks, processes, systems, methods or packages generate numbers of some sort - scores, ratings or whatever - measuring the risks.  We're not going to delve into the pros and cons of various RA methods here, nor discuss the differences between quantitative and qualitative approaches.  We know some methods only go as far as categorizing risks into crude levels (e.g. low, medium, high, extreme), while others produce percentages or other values.  We assume that ACME has chosen and uses one or more RA methods to analyze its information security risks, and on the whole management finds the RA process useful.

  

The question is: are information security risk scores produced by RA valuable as an information security metric?

P	R	A	G	M	A	T	I	C	Score
72	60	55	70	71	40	60	60	50	60%

In the view of ACME's management, the RA scores have some merit as an information security metric.  They do quite well in terms of their Predictiveness, Genuinness and Meaning, but there are concerns about their Actionability, Accuracy and Cost-effectiveness.   The overall PRAGMATIC score is somewhat disappointing.

If ACME management definitely wants to measure information security risks but there are no higher-scoring metrics on the cards, they have a few choices.  They might:

• Accept the metric as it is;
• Weight the PRAGMATIC ratings to emphasize the factors that are most important in this specific area (e.g. Predictiveness and Relevance), then re-compare the scores and reconsider the candidate metrics; 
• Adopt this metric as a temporary measure for now but, while gaining experience of the metric, actively search for something better;
• Reject the metric and carry on searching for something better;
• Make changes to the metric in order to address its PRAGMATIC weaknesses, hopefully without compromising its strengths;
• Conduct a trial, comparing a few metrics in this area, including variants of this one, over the course of a few months;
• Reconsider what it is that they really want to know and try to be more explicit about the goals or objectives of measurement in order to prompt the selection/design of better metrics candidates;
• Review the PRAGMATIC ratings for this and other risk-related metrics, challenging their assumptions and considering more creative approaches;
• Adopt complementary metrics or use some other approach to compensate for the weaknesses in this metric.

Risk matrix

I developed this 3x3 matrix today as part of an information security awareness module about risk management.  The matrix plots the severity (or impact or consequences or costs) of various kinds of information security incident against the frequency (chance, probability or likelihood) of those same kinds of incident.  It is color coded according to the level of risk.

Clearly, the incidents shown are just a few illustrative examples.  Furthermore, we could probably argue all day about their positions on the matrix (more on that below).

Some might claim that it doesn't even qualify as a metric since there are no actual numbers on the matrix.  "Cobblers" I say!  It is information that would be relevant to decision making and that's good enough for me, but if you feel so inclined, go ahead and pencil-in some suitable numbers at the boundaries of those severity and frequency categories ... and be prepared to argue with management about those numbers as well!

Anyway, it set me thinking about whether the matrix might form the basis of a worthwhile information security metric.  The obvious approach is to score it on the PRAGMATIC scale, in the context of an imaginary organization ("ACME Enterprises" will suffice):

P	R	A	G	M	A	T	I	C	Score
70	70	50	65	70	40	55	45	85	61%

The 61% score implies that this metric has some potential for ACME, but it doesn't stand out as an must-have addition to their information security measurement system.

There is a question mark about its Independence since the people most likely to be preparing such a matrix (i.e. information security pros like me) generally have something of a vested interest in the risk management process.  

I wouldn't exactly call it Actionable since it doesn't directly indicate what we need to do to reduce the severity or frequency of the incidents ... but it does at least show that reducing either aspect will reduce the risk.  It is Actionable in the sense that preparing and discussing such a matrix would be a useful part of the risk analysis process, particularly if the discussion engaged other risk professionals and business managers (which would also increase its Independence rating, by the way).

Involving managers in the process of preparing, reviewing or updating the matrix would increase the cost of the metric, but at the same time would generate more value, so the Cost-effectiveness would end up more-or-less unchanged.  

I mentioned earlier that we might 'argue all day' about the positions of incidents on the matrix, but in so doing we would be considering and discussing information security incidents, risks and controls in some depth - which is itself a valuable use or outcome for this metric.  Understanding the risks would help management prioritize them and, in turn, allocate sufficient resources to treat them sensibly (for example investing in better malware controls rather than, say,  expensive state-of-the-art CCTV system to reduce a risk that is already in the green zone).

So, all in all, it's an interesting security metric although it needs a bit more thinking time yet ...

UPDATE: there is a remarkably similar metric at the heart of the "Analog Risk Assessment" ARA method I describe here.

Measuring things right

A random comment of unknown origin fluttered into my consciousness today:

“Made with the highest attention to the wrong detail”

It set me thinking about the waste of effort that goes into oh-so-carefully measuring and reporting the wrong things, for reasons that include:

• Failing to determine the information actually required, and/or mistakenly assuming the nature of the inquiries (and, perhaps, reporting to the wrong audience)
• Naivete and lack of understanding about metrics, measurement, decision making and/or statistics in general
• Using certain measures simply because the base numbers and/or the charts, tables and reports are readily available (so they must be useful, right?)
• Presuming the need for a high level of accuracy and precision when in fact rough-and-ready indicators would be perfectly adequate (and cheaper) (and quicker)
• Analytical errors e.g. believing that the measured item is well-correlated with or predictive of something of interest when in fact it isn't
• Brain-in-neutral: no discernable thought patterns or reasoning behind the choice of metrics, at least nothing concrete that we can recall if pressed
• Falsely equating voluminous data with information or knowledge (knowing everything about nothing)
• Adopting metrics used, recommended, mentioned or discussed by others in conversations, articles, standards, metrics catalogs, websites or blogs (yes, including this one!) without considering one's own information needs and the differing contexts
• Giving the appearance of Being On Top Of Things ("The value is 27.435 ... but ... OK I'm not entirely sure what that means")
• Generating chaff, deliberately distracting the audience from genuine issues by bombarding them with spurious data, theories, arguments and presumptions
• Being "too clever by half" - an obsessive/compulsive fascination with particularly complex or obscure yet strangely intriguing metrics having a whiff of magic
• Being required to spout nonsense either by some authority who perhaps doesn't understand the issues but wants to be seen to be Doing Something, or in accordance with a poorly-considered contract, Service Level Agreement or reporting system
• Continuing to use (= failing to challenge and revise/withdraw) old metrics long after they should have been dispatched to a rest home for the sadly bewildered
• Re-using metrics that have proven worthwhile elsewhere/in other contexts on the mistaken assumption that they are 'universal'
• Filling spaces on a fancy dashboard or management report with "pretty" metrics (eye-candy), triumphs of form over substance
• Desperation stemming from an apparent lack of alternatives due to limited capabilities and skills, a lack of imagination/creativity and/or no will or opportunity to design or select more suitable metrics

That's some list!  If I'm honest, I am personally guilty of some of them.  I've been there, done that, and still I see others treading the same route.  I think I'm wiser now, but only time will tell if the effort required to think more deeply about metrics and write the book has led to a breakthrough, or whether (or, more accurately, in which respects) I am still deluding myself.

Think about this issue the next time you find yourself poring over a management report or survey, trying to make sense of the content.  Even more so if you are skimming and discarding metrics without making any real effort to understand them.  Ask yourself whether you actually need to know whatever it is you are being told, and why it concerns you.  In GQM parlance, consider the Goals and the Questions behind the Metrics on the table.  Understand also that there may be hidden agendas, false assumptions and plain errors in the numbers.  Consider the methods used to gather, analyze and present the metrics, including their mathematical/scientific validity (e.g. was the population sampled randomly or selectively?  Is the variance significant?  Are there uncontrolled factors?).  Be dubious, cynical even, if it makes you contemplate the true meaning and worth of the information presented.  

For therein hides genuine insight.

Metric of the week #48: redundant controls

Security Metric of the Week #48: proportion of information security controls that are ossified

Information security risks and controls change gradually within the organization.  From time to time, new risks emerge, new controls are introduced, existing controls find new applications, old risks subside and redundant controls are retired from service - at least in theory they are.  Keeping the portfolio of controls aligned with the risks is the primary purpose of the information security function and the Information Security Management System.  This week's metric is one way to measure how well that alignment is being managed in practice. 

In immature organizations, security controls once accepted and installed seem to become permanent fixtures, integral parts of the corporate infrastructure.  Successive risk analyses lead to the introduction of yet more controls, but nobody ever quite gets around to reviewing and thinning-out the existing corpus of controls.  Gradually, controls accumulate while the business becomes ever more constrained and inefficient, carrying a costly burden of redundant, unnecessary and unsuitable controls.  

This is understandable and perhaps appropriate during the early growth phases of information security/risk management when the focus is on bringing the organization up to an acceptable baseline level of security by implementing basic controls.  After a few years, however, there is a good chance that some of the installed controls are no longer needed.  It's not unusual to find a few controls that nobody understands, needs or supports - their original purpose has long since been forgotten and may in fact no longer exist (e.g. compliance obligations that no longer apply).  Some will have been superseded by other controls, changes in the business, and changes in the risks.

The metric implies that someone has to count information security controls, and determine how many of them are ossified, redundant and no longer necessary.  Counting controls is easier said than done: for example, is every PC running the corporate antivirus software package counted separately?  Is the antivirus package a single control, since it is probably addressing several distinct malware-related risks and perhaps others such as spam?  This is something that would need to be sorted out when specifying the metric in order to ensure that successive readings were directly comparable, since this metric is about the medium to long term trend rather than point values. 

According to the management at ACME Enterprises Inc., here is the PRAGMATIC rating for this metric:

P	R	A	G	M	A	T	I	C	Score
85	88	85	80	84	75	22	62	39	69%

They are evidently most concerned about the metric's Timeliness and Cost-effectiveness.  Counting and assessing the status of controls is undoubtedly a tedious, time-consuming process: management is not sure it is worth the effort.

In discussing this metric, the ACME's Information Security Manager acknowledged that she ought to be actively looking for ossified controls.  She argued that this was an operational security management activity under her remit, and she did not require a metric, preferring instead to put her efforts into systematically addressing the ossified controls as they were identified.  On the other hand, the Chief Information Security Officer sought reassurance that the process was, in fact, operating well, and felt that the metric would be a useful way to demonstrate to more senior management that information security was playing an active part in cost containment.

As we go to press, the discussion is unresolved.  Other candidate metrics are being compared on the basis of their PRAGMATIC scores, while the ISM and CISO are exploring their requirements in more depth.  If you were advising them on their metrics, what would you suggest?

Website attack exposure metric

Page Rank versus Internet attack rate - a worked example of the metrics selection/design process

In theory, organizations that establish a substantial web presence for marketing reasons are also, potentially, setting themselves up as high-profile targets for Internet-based attacks.   But is the premise true?  Are organizations that maintain high-profile websites attacked more often from the Internet than those that maintain a low profile?  Let's try to address the question by developing a metric using the PRAGMATIC approach.

Most websites are routinely scored or ranked by Internet search engines and social networking sites.   Some popularity measures are distinctly dubious, however, but Google Page Rank is a simple, widely-used and well-respected measure of the visibility or popularity of a website, so that part of the equation is easy enough. 

To measure Internet-based attacks, we might count up the number of web-related information security incidents such as website hacks, worms, social engineering/frauds etc. over a suitable period (e.g. an hour, a day, a week or a month), using log and alert data from the organization's firewall, intrusion detection system, incident reporting/management system and so on.   Although this information will take more effort to gather than Page Rank, it's feasible provided the numbers are readily available from the respective security systems.

So, if we now divide Page Rank by weekly attack count, we have our metric ... but in order to answer our rhetorical question we need to compare organizations on a comparable basis.  Here we run into two problems: 

1. Although the algorithms are proprietary to Google, we understand that Page Rank is determined mechanistically from Google's search data, hence different websites can presumably be compared directly by their Page Ranks.  The rate of Internet-based attacks, however, is not well defined.  Even if we standardize the measurement period or adjust the rates accordingly, different organizations have different firewalls, different working definitions of Internet attacks, different detection sensitivity and so on.   It's going to be tough to standardize or normalize all these elements.
2. Page Ranks are available in the public domain, but Internet-based attack rates aren't.  Most organizations consider information security-related data sensitive, if not secret.  The information might be available under some sort of non-disclosure agreement, or via a trusted intermediary such as the bodies that routinely survey and compare organizations' security for benchmarking purposes (e.g. the Information Security Forum).  Organizations that choose not to share their information cannot be used by others as comparators.

At this point, let's see how the metric rates using PRAGMATIC:

• Predictiveness: provided the premise is true, an increase in Page Rank will probably increase the rate of Internet-based attacks, and an organization with a lower PR than a peer might anticipate a lower rate of Internet-based attacks.  However, the premise is unproven and a significant proportion of Internet-based attacks are non-specific and undirected - automated scans for vulnerable software, worms, spam etc.  So let's give it 45% for this criterion.
• Relevance:  although Internet attack rate is clearly of interest to those managing, directing and funding Internet-related information security, we don't actually know whether the Page Rank element of this metric is relevant to information security or not: that's the rhetorical question we set ourselves earlier.  We think it is, but we're not sure.  Score 65%.
• Actionability: here we have a BIG problem.  What are we supposed to do with this metric?  Let's say the number is 'very high' or 'very low' - are we expected to do something either way, and if so what?  There's not a lot we can do about our Page Rank nor the rate of Internet-based attacks against us.  We could tweak the web marketing budget, maybe, and increase or decrease the deterrents against attacking us, but to be honest neither one is very effective.  Score: 20%
• Genuinness: how hard would it be to fake or manipulate this metric?  The Page Rank part is pretty much beyond our control, but we already noted that there are various elements to Internet attack rate, so there is some interpretation required there and the potential for someone so inclined to be deliberately selective when generating the numbers.  Score: 75%
• Meaningfulness:  this criterion specifically concerns the perspective of the intended audience of the metric.  Oh oh!  Up to this point, we haven't really considered who that might be.  Who might be interested in the number?  Who might use the information?  The CISO or Head of Marketing maybe?  It's not entirely obvious.  Given that it has taken us so many words to explain this metric to you, and given that you are reading an infosec metrics blog, you are probably more clued-up than the audience for the metric, this metric clearly does not qualify as "self-evident" either.  Score 25%
• Accuracy: whereas the attack-rate part of the metric is very granular, Page Rank has just nine possible values, ten if there is such a thing as a PR0 page.  Page Rank applies to individual pages not whole websites, so we could check the PR of several pages and generate a mean, I guess, but I doubt it is worth the effort.  The PR of the home page is probably "close enough for Government work" but perhaps you ought to check if you decide to go ahead with this metric.  Score 45%
• Timeliness: determining the attack rate involves collating statistics from various sources, but it's not unreasonable to assume that most if not all of them are automated, hence once the appropriate data feeds are configured, the metric should be available pretty much immediately at the end of the reporting period.  Furthermore if we adjust something as a result of the metric, we would expect to see a change in the numbers within a few reporting periods.  Score 80%
• Independence: Page Rank comes from Google, and can be verified easily.  The other figures come from the security systems and could also be verified with a bit more work, but there is a residual risk that someone might manipulate the systems, the figures or the calculations for their own purpose (whatever that might be!).  Score 80%
• Cost-effectiveness: the metric is fairly cheap to generate, once the security systems have been correctly configured.  Changes to the systems would require changes to the data collection and calculations, and probably some effort to re-base prior periods if trends are important, but again these are not especially costly.  However, the other part of this criterion is the business value of the metric, and there we struggle.  It's not clear who would be the audience for the metric, nor how they would use it, consequently the business value is hard to fathom.  Score 20%

The overall PRAGMATIC score, then, is a simple unweighted mean: 51%.  The metric as currently described has little to commend it.  However, the analysis has pointed out a number of issues that might be addressed during the metrics design process, in particular clarifying its audience and purpose.  Unless we make genuine headway on those factors, there are probably other more PRAGMATIC metrics to keep us occupied.

To conclude this worked example, three Hinson tips to mull over at your leisure:

1. You might quarrel with those PRAGMATIC ratings, in fact I would be very disappointed if you didn't because that would imply that you haven't fully engaged with the process.  The ratings are subjective and as such there is definitely room for interpretation, discussion, argument even.  In this context, that's a very positive thing because it forces us to think through the issues and elaborate on our requirements, concerns and thoughts, considering each other's perspective and developing our own.  The final score has some value but the thinking behind it is invaluable.  Done well, the PRAGMATIC method involves active participation and, yes, the odd laugh.  
2. Astute readers of our book, or of Lance Hayden's, may have noticed that we skipped the G in the classic GQM approach - we didn't clarify the business Goal but leaped directly into the Question and then the Metric.  Tut tut, go to the back of the class.  This is the root of the issue of the unclear audience and purpose for the metric.
3. We selected a random graph to illustrate this blog item, but if we were serious about proposing this metric, it would be well worthwhile mocking-up the metric to see how it might look if we went ahead with it.  Doing so would force us to think about the type and style of graph, the axes including the timescale, and the data points to be plotted.  It would also give us something more concrete to discuss with our potential audience, a focal point (and, yes, we'd have to discuss it with someone, so that would be our cue to decide who!).  Even better would be to discuss a small sheaf of alternative metrics, considering their PRAGMATIC scores on a relative basis and gradually figuring out their pros and cons, enabling us to shortlist and refine a single metric, or maybe a couple.  Less is more.

SMotW #47: inactive user accounts disabled

Security Metric of the Week #47:  proportion of inactive computer user accounts disabled in accordance with policy

To calculate this metric, someone first checks how many inactive user accounts there were on the systems in total, and then how many of them were disabled as they should have been according to ACME's policy during the reporting period (e.g. this calendar quarter).

Counting inactive accounts is tedious if it involves manually checking activity records maintained by the individual IT systems, but easier if the checks can be performed by running scripts on a limited number of shared/centralized network user authentication systems (such as domain servers for Windows domains), in which case this becomes a fairly straightforward and useful compliance measure.

ACME's management determined the following PRAGMATIC numbers for this metric:

P	R	A	G	M	A	T	I	C	Score
68	56	74	76	73	64	64	52	75	67%

The PRAGMATIC ratings are fairly flat across the board - in other words, the metric is neither particularly strong nor weak in any factor.  Management had some concerns about its Relevance to information security since it is such a narrow technical metric, and also about the Independence factor since those who are most likely to be measuring and reporting it have a vested interest in the metric.  On the other hand, they appreciated the fact that it is quite Genuine (readily verified or audited if needs be) and Cost-effective since ACME has a centralised system for user account management. 

Disabling computer accounts when employees leave the organization is a rather basic IT security control.  If this is not happening reliably, there are probably even more serious issues with the user administration processes, and in fact with information security controls in general - in other words, the metric could be taken as a rough indicator of ACME's overall information security status.  This boosted its Meaningfulness rating. 

Furthermore, the rate of improvement of the specific metric is also a rough indication of the extent to which ACME's managers are truly managing, controlling or influencing the organization's information security as a whole: if it takes several months of effort to get the proportion of inactive accounts disabled up from, say, 10% (dreadful) to 30% (poor), this is clearly worse than if the same improvement happens in a matter of days or weeks, or if the metric reaches 75% or more during the same period.

Talking of period, the frequency at which the metric is reported is another measurement parameter that management can adjust.  They may, for instance, ask for more frequent updates on the metric while the control is being targeted for improvement (perhaps once a month), then revert to quarterly or whatever when sufficient improvement has been made.  Another option would be to stick to quarterly reporting for senior management but track and report the status monthly or even weekly for operational managers who are actively working to improve user account administration.