SMotW #78: % of policies addressing viable risks

Security Metric of the Week #78: proportion of information security policies addressing genuine risks

Writing policies is easy: writing policies that are relevant is harder than it sounds, and writing policies that are readable, fully implemented and straightforward to assess for compliance is harder still.  This week's example metric is intended to highlight policies that are irrelevant or pointless, for example those that concern situations or risks that are remote (extremely unlikely) in practice.  If there is no conceivable combination of threat, vulnerability and business impact, there is little chance of an information security incident, hence a policy (or in fact any other form of control) is probably unnecessary.  

ACME rated the metric at 75% on the PRAGMATIC scale:

P	R	A	G	M	A	T	I	C	Score
65	76	91	73	83	77	70	61	78	75%

There was some interest in using the metric to identify unnecessary policies and controls that could be eliminated, simplifying ACME's Information Security Management System, cutting costs and red tape while at the same time focusing on the remaining policies and controls - the ones that actually matter. 

This is an example of a management-level metric, as opposed to those of strategic value to senior management or more operational in nature.  It's the kind of metric that ACME's Information Security Manager might generate and report periodically to the CISO while sifting systematically through the policies and controls, weeding-out any that are no longer relevant.

On the matter of policies being 'readable, fully implemented and straightforward to assess for compliance', ACME considered other metrics that would complement this one.  It might be possible to come up with one security metric that covers all policy aspects, and while senior management might prefer a single high-level metric, it would probably be based on a variety of more detailed metrics such as this one that are more useful to the CISO, ISM and others.  

SMotW #77: Computer suite power vs aircon

Security Metric of the Week #77: Computer suite power consumption versus its air conditioning capacity

Monitoring the total electrical power consumed by the computer suite (building, room, cupboard, whatever) is a basic control measure that is useful for managing the power supply, for example ensuring that it remains within the safety limits and engineering constraints of the cabling and switch-gear. If (when!  Hopefully before!) the power consumption approaches some limit, decisions have to be made about leveling off the trend (e.g. by replacing older IT equipment with more energy-efficient green stuff) and upgrading the supply, preferably through a coherent, planned, professional program of engineering work. Computer suite power consumption is itself a highly PRAGMATIC metric. 

In addition to energy from the environment and sunshine, a significant proportion of the electrical energy pumped in to the computer suite through the power system turns into heat that also has to be removed by the air conditioning system.  Therefore, the power consumption is closely associated with the air conditioning load, which in turn has implications on the capacity and reliability of the aircon, which obviously affects the computer equipment and ICT services supplied.  

Measuring the heat load in the computer suite is trickier thank one might suppose. It is feasible to maintain an inventory of the installed equipment, with details about the power consumption of every item, but it takes some effort to keep it sufficiently complete and accurate. Periodic audits are probably going to be needed if you go down this route. The inventory is therefore a fairly costly control, but the expense may be justified if the inventory is sufficiently valuable. In addition to the power consumption and heat load aspect, an inventory is also useful for capacity planning, systems and network management, equipment configuration and maintenance, software licensing, business continuity management, and financial management.  

In contrast, measuring the actual power consumption is an easy option: even a simple clamp-on ammeter will tell the electrical engineers roughly how much power is being consumed on each circuit at an instant, while computerised full-time power logging and alerting is no big deal.  [The data may even be precise enough to enable someone to cross-check the inventory if there are sudden changes in power consumption, for instance when someone installs a new system but forgets to update the records!]

Comparing the power consumption - energy input - against the air-conditioning installed and operating capacity - energy output - is one way to confirm that the air-conditioners are not being pushed too hard, supplementing other measures such as the most obvious one, room temperature. 

Tracking and comparing both parameters over time will indicate whether the tendency to install more and more powerful IT equipment is being matched by corresponding changes to the power and cooling systems. It will inform important decisions about maintenance and upgrades.

ACME managers rated the metric thus:

P	R	A	G	M	A	T	I	C	Score
81	69	89	92	80	99	98	90	98	88%

The excellent ratings for Accuracy, Timeliness and Cost-effectiveness push this metric high on ACME's metrics wish-list.  

Measures of this nature are valuable for most organizations, and are more-or-less essential for organizations that depend heavily on the availability of critical IT systems. The metric is, in fact, an example from an entire category of physical/engineering-type information security metrics that is often neglected, particularly when the physical facilities are managed by a corporate or in/outsourced function independently of IT. Think about it: do you have the information to know whether the UPS and generators supporting your mission-critical computer facilities are operating comfortably within their limits, or are they running on a knife edge, ready to fall in a heap if not burst into flames the next time someone plugs in yet another ultra high capacity blade server into a bulging rack? Is your computer equipment about to fall through the false floor, literally, or choke itself up on dust? Are the fire alarms and extinguisher systems being professionally maintained, adapted and tested to reflect changes in the layout and use of the racks? There are substantial information security risks associated with the physical environment and supplies for the computer suite: are your metrics up to scratch?

Furthermore, this metric is shining example of the value of analyzing suitable combinations of raw data or information from independent measures, gaining additional insight over and above considering each one in isolation. While ISO/IEC 27004:2009 blabbers on about 'base measures', 'derived measures' and 'indicators' in its inimitable style, this is a practical illustration of the concept. The same principle of aggregating, comparing and contrasting measures applies at even higher levels too, so in theory it would be possible to end up with the ultimate measure-of-measures, an overall "information security score" for the enterprise. As to whether that is possible or sensible in reality is left as a little metrics exercise for you, dear reader.

SMotW #76: Policy quality

Security Metric of the Week #76: quality of information security policies

Poor-quality, badly-written policies:

• Are less likely to be understood by the people they are meant to guide/direct/control;
• May neglect, under-play or mis-state certain obligations or requirements;
• Are probably not being actively maintained and so are probably outdated;
• May conflict with other policies, plus strategies, standards, procedures, guidelines, laws, regulations etc.;
• May create legal liabilities or open up opportunities for people or organizations to claim they don't apply, are unworkable, are unreadable etc.;
• May create excessive, unnecessary or inappropriate requirements and costs (red-tape);
• Give the appearance of expressing management's intent but in fact probably do not (fully) address the information security issues/risks they are supposed or intended to address;
• Don't work - compliance is likely to be low, they are more likely to be flaunted or openly ignored;
• Reflect badly on the professionalism of their authors, and on the value and importance of the information security program as a whole;
• Indicate a lack of concern and support for information security by management, bordering on carelessness, neglect or disrespect, most likely reflecting a serious underlying governance issue.

Those are pretty severe consequences if you think about it. Policies, along with the associated policy lifecycle management/maintenance processes, authorization, awareness/training, compliance activities etc. are therefore an important element of the organization's information security. No wonder they are explicitly required in clause 5.2 of ISO/IEC 27001:2013 and section 5 of ISO/IEC 27002:2013, as well as most other information security standards and frameworks and even some laws.

As a consultant, I can tell a lot about an organization's information security status simply by reviewing its policy-related materials. The quality and style of the writing says as much as the literal content. An hour or three spent in a quiet corner reading and contemplating the policy stuff will answer questions such as:

• Do the policies read well? Are they straightforward, easy to understand, well-structured, sensibly laid out? Is the grammar and spelling OK? Is there a standard, recognizable structure and layout for all of them?
• Are they relevant, interesting even? Do they bother to explain why they exist, the context, background and purpose of the controls laid out? Do they make a case for compliance, or simply insist?
• Is there evidence of a grand design - a coherence and fit between related policies? No significant overlaps or gaps? No bald conflicts or nonsense? Do they follow a recognized structure or framework (such as ISO27k, COBIT, SP800, BMIS ...) or does it look like someone has been busy reinventing the wheel?
• Are they pragmatic, workable, usable ... or are they theoretical, unduly formalized, stilted and essentially unworkable? [Hint: legalese, pseduo-legal jargon and archaic terms such as "including but not limited to" and "heretofore" are big flappy red flags as far as I am concerned. Such language usually indicates that someone who is NOT legally-trained has been in charge or involved in the writing, and has tried to appear clever. If your corporate lawyers or other professionals actually insist on incorporating that kind of tripe in your policies, you have my deepest sympathies for the constraint.]
• Does it appear that they are being actively maintained? Are they up-to-date with current risks and controls? Is there a nominal owner, someone with sufficient seniority and gravitas to make people at all levels (including senior management!) aware that they are obliged to comply? Is there evidence of a timely review and (re-)authorization process? Is there version control and a history of changes?
• Is there evidence that they are taken seriously e.g. formal authorization or mandate, stated applicability and scope?

The icing on the cake for me includes things such as stating compliance obligations plus related roles and responsibilities, associated metrics, and explicit cross-references to related policies, procedures, standards, laws, regulations etc., including those that fall outside the remit of the Information Security Management function (e.g. HR policies, legal/regulatory compliance policies, risk management and audit policies) ... but, at the same time, I give bonus marks for conciseness and clarity!

Anyway, the green pie chart demonstrates the kind of metric that ACME Enterprises had in mind when their managers were assessing and scoring this as a potential information security metric. The idea is for someone appropriate to review and classify the policies (and perhaps related materials) using criteria derived along the lines of the notes above. ACME's PRAGMATIC ratings for this metric were as follows:

P	R	A	G	M	A	T	I	C	Score
80	85	40	66	72	75	80	80	80	73%

With most ratings hitting the 70s and 80s, the only real fly in the ointment was Actionability at 40% since the pie chart does not indicate why certain policies were deemed of low quality, nor which policies were in the lower quality categories, hence it is not readily apparent what needs to be done to improve the metric. It's obvious how that issue could be addressed (asking the assessor/s for more information) for an even higher score, making this security metric a strong candidate for ACME ... and, we suggest, for almost every organization, given the aforementioned importance of security policies.

And yes, 'aforementioned' is one of those archaic words I despise!

Giant Green Land Snails

A fascinating thread is being woven over on the ISO27k Forum, concerning information security risk analysis (RA) methods. Bob Ralph made a good point that set me thinking this morning: 

"... 'unknown knowns' should be scored the very highest until you know otherwise ... Of course the 'unknown unknowns' will not even be on the RA. But they are still about somewhere."

Biologists have developed techniques for estimating the unknown and answering awkward questions such as "How many Giant Green Land Snails are there on this island?" The obvious technique is to try to catch and count them all, but that's (a) costly, (b) disruptive for the snails and their ecosystem, and (c) not as accurate as you might think (snails are well camouflaged and duck for cover when biologists approach!). Capture-mark-recapture, also known as tag-and-release, is a more useful technique: catch all the snails you can find in a given area, mark or uniquely identify them in some way (preferably not with bright yellow blobs of paint that make them even easier targets for the Little Blue Snailcatcher bird that preys on Giant Green Land Snails!), then some time later repeat the exercise, this time counting the number of snails that are marked/identified or new. From that proportion, estimate the total population of Giant Green Land Snails in the area, and extrapolate across the entire island, taking various other factors into account (e.g. nesting areas for the Little Blue Snailcatchers, quantity and quality of habitats for the snails, snail lifetime, foraging range etc.). There are statistical techniques supporting this kind of method, and various other methods that give reasonable estimates, sufficient to answer the original and related questions, such as "Is the population shrinking or expanding?". I'm sure there are similar approaches in other fields of science - estimating the age/size of the universe for instance.

Go back to the last paragraph to swap "hackers" for Giant Green Land Snails, and "law enforcement" for biologists, and you have a technique for estimating the size (or, in fact, other characteristics) of the hacker population. It's "only" an estimate, but that is better than a pure guess since it is based on measuring/counting and statistics, i.e. it has a scientific, factual, reasonably repeatable and accurate basis. Douglas Hubbard's excellent book "How to measure anything" talks at length about the value of estimation, and (in some circumstances) even what one might call WAGs (wild-arse-guesses) - it's a very stimulating read.

So, I'm thinking about how to apply this to measuring information security risks, threats in particular. We have partial knowledge of the threats Out There (and, to be accurate, In Here too) gleaned from identified incidents that have been investigated back to the corresponding threats. There are other threats that are dormant or emerging, or that are so clever/lucky as to have escaped detection so far (Advanced Persistent Threats and others). There are errors in our processes for identifying and investigating incidents (meaning there are measurement risks - a chance that we will materially miscalculate things and over- or under-estimate), and a generalized secrecy in this field that makes it tricky to gather and share reliable statistics although some information is public knowledge, or is shared within trusted groups. But the overall lesson is that the problem of the "known and unknown unknowns" is not intractable: there are data, there are methods, there is a need to estimate threats, and it can be done.

One thing the scientists do but we information security bods don't (usually) is to calculate the likely errors associated with their numbers. So, in the snail example, the study might estimate that "There are 2,500 Giant Green Land Snails on the island, with a standard deviation of 850" or "We are 95% certain that the total population of Giant Green Land Snails on the island is between 420 and 735". There are numerous situations in information security in which errors or confidence limits could be calculated statistically from our data, but we very rarely (if ever!) see them in print - for instance, in survey-type studies where there are sufficient people or organizations in the pool for the statistics to work out (and with the right statistics, a surprisingly low minimum sample size may be sufficient, less than the 30 that used to be our rule of thumb). 

Speaking as a reformed (resting, latent, recuperating, ex-) scientist, current real-world information security practice is largely unscientific, outside of academic studies and journals anyway. Aside from the issue just mentioned, surveys and other data sources rarely explain their methods properly - for instance they may (if we're lucky) mention the sample size, or more often the number of respondents (a different parameter), but seldom are we told exactly how the sample was selected. With vendor-sponsored surveys, there is a distinct possibility that the sampling was far from random (e.g. they surveyed their own customers, who have patently expressed a preference for the vendor's products).  Small, stratified and often self-selected samples are the norm, as are implicit or explicit extrapolations to the entire world.

Consequently, for risk analysis purposes, we are often faced with using a bunch of numbers of uncertain vintage and dubious origins, with all manner of biases and constraints.  And surprise surprise we are often caught out by invalid assumptions. Ho hum.

Regards,

Gary 

PS  The snails and birds are pigments of my imagination.

Expressing security metrics in business terms

On the Security Leadership Solutions Executive Council Faculty Advisor blog, Kathleen Kotwica raised a good point about ineffective security metrics:

"The issue many security practitioners incur is a) not measuring at all or b) measuring things by simply counting them (e.g., workplace violence incidents or lost laptops), rather than demonstrating the value Security brings to the business. By way of example, convey savings to the company by your program's reduction of workplace violence issues. That is, the cost of managing an event and lost employee time; or cost savings by reducing any potential acts because of your background due diligence program."

Although the blog concerns physical security metrics, Kathleen's point applies equally to the broader class of information security metrics.  And the issue is not simply about counting things, but about presenting simple counts, or indeed other statistics, without giving them relevance and meaning to the business.

Suppose, for instance, I include the number "27" in a management report. The bare number, by itself, is meaningless, begging the question "27 what?" The 'what' is the unit, for example "27 security vulnerabilities." That may be a valid data point but it barely qualifies as information without additional context, and it's a long way from providing worthwhile knowledge and insight that will help run the business.

One way to enrich a metric is to indicate the trend, for example "This reporting period, the number of security vulnerabilities fell from 29 to 27" ... mmm, better but still rather obscure. A reduction in the number of vulnerabilities sounds like it is probably a good thing, but the reduction from 29 to 27 is hardly a dramatic fall ... unless it has been stubbornly stuck at 29 for ages, or had been increasing over prior periods.  Plotting (graphing) the number over time is the simplest way to show trends, while we can get creative with the axes, annotation and colors to add yet more information, so long as it actually helps.  [The gaudy figures we use to illustrate this blog are deliberately intended to catch your attention and fire up your imagination: our style may look out of place in most internal corporate reports, but why does everything have to be so drab and lifeless? Banish monochrome dullness!]

Annotations can be text pointing out interesting, relevant features directly printed on the graph, or brightly colored numbered/lettered blobs on the graph linking to a separate panel of text, or better still a presenter physically pointing to items of interest and discussing them. Textual descriptions are usually required for any specialist terms on the graph, not least the title or caption. 

The wording of the annotations/text is very important, yet few statistics or metrics books go into this aspect. Compare, for instance "Vulnerability Reduction Project comes into effect" with "Vulnerabilities fall at last!" The former is informative but flat, while the latter makes a more emphatic statement, which may or may not be appropriate. These are extremely trivial examples, but I'm sure you can think of real-life situations where suitable or unsuitable text would make a huge difference to the way the same metric is perceived. Look through any security survey for examples of how to point out and expand on the numbers. It takes skill to do this well, without misleading or boring the pants off the audience and none of us gets it 100% right - not least because 'the audience' is often a wide range of people with widely differing perspectives and information needs. Some want just the facts, others need to be led through the analytic thinking process, perhaps as far as pointing out the key conclusions for them.

Getting back to Kathleen's point, expanding on the business implications of the numbers is generally a sensible move for management audiences, which means pre-considering the obvious response "So what?" Why should anyone care that the number of vulnerabilities is 29? What does that figure say to those making decisions about security risks and controls? If you can't answer such questions convincingly in a way that makes sense and motivates the audience to do something appropriate with the information you are providing, then you should not be surprised if the security metric has no appreciable effect on your universe. And don't be upset if I suggest that the metric is a waste of valuable space.

Consider, also, turning this entire discussion on its head. Start by figuring out the issues that need to be addressed, and the questions that doing so will raise, then generate the metrics accordingly to address the questions. Make the entire metrics selection/design process business-led from the outset, rather than attempting to justify your metrics in business terms after the fact. That way, security metrics have a clear purpose, so the context and relevance are easy to determine - in fact, if highly PRAGMATIC metrics may not need to be 'explained in business terms' in an explicit fashion, since to a large extent they will be self-evident, obvious even. The Actionability and Meaningfulness of a metric are specifically assessed for this very purpose, while Relevance and the remaining PRAGMATIC criteria play their parts too.

SMotW #75: noncompliant controls

Security Metric of the Week #75:  number of controls failing to meet defined control criteria/objectives

The premise for this metric is that information security controls are, or rather should be, designed to achieve something fairly specific i.e. the control objective. Provided they are well worded, it ought to be possible to assess the extent to which control objectives are satisfied in order to determine whether the corresponding controls are adequate.

The PRAGMATIC ratings for this metric, according to ACME's managers, are:

P	R	A	G	M	A	T	I	C	Score
88	86	88	65	78	60	26	90	70	72%

The metric's Timeliness rating depends on the testing-and-reporting period. The gaudy traffic-light colored example graph above shows a monthly reporting period which would be good for timeliness, but assessing and reassessing this metric every month would be beyond the limited capabilities of ACME, at least not without seriously impacting the metric's Cost-effectiveness and Independence: they had in mind asking Internal Audit to measure and report the metric annually, a rather different basis.

ACME management perceived this metric as a way to drive a long-term process of matching controls to control objectives to business outcomes. That was their mind-set when they scored it.

You, however, may score this metric quite differently, particularly if your understanding of (a) what the metric is actually measuring and (b) how it is to be measured, differs materially from ACME's - and that strikes at the core of an important issue. At face value, it would make sense for organizations to share their best security metrics with each other. The trouble is that, just like 'best practices', 'best security metrics' is a heavily-loaded term. What's best for you may not be best for me, and vice versa.

The PRAGMATIC approach does at least give us a mechanism to consider and discuss the pros and cons of our metrics, and even to compare them on a similar basis, but the organizational context in which the metrics are to be used is crucial to the choice. 

We discuss at some length in the book the question of what metrics are for - the questions or issues that security metrics meant to address. One might argue that at a high level, most organizations have broadly similar goals (such as 'satisfy market demand' and 'make a profit') but it is not obvious how to derive information security metrics from those. Analysis in more depth will tease out information security objectives and suggest relevant metrics, but aside from the facile 'protect information', the details vary between organizations, and hence different metrics are probably needed.

What's more, the security metrics requirements within a single organization are likely to change over time for two distinct reasons:

1) The organizational context is constantly evolving. Management may, for instance, have a pressing need for compliance metrics in the run up to major compliance deadlines or audits, but at other times compliance may be off the radar, replaced by the drive for continuous improvement or incident management or continuity planning or governance or ... whatever;

2) The information security measurement system, as a whole, evolves, as does the information security management system that it supports. The issues that get translated into metrics when an organization is first getting to grips with its security metrics may be long forgotten a few months or years down the track, particularly if the metrics were successful in helping management resolve those issues.

The upshot is that there is less to be gained from sharing metrics than some would have you believe. Sharing the PRAGMATIC approach, now that's a different matter entirely. 'Give a man a fish and you feed him for a day. Teach a man how to fish and you feed him for a lifetime.'