Enterprise Security Metrics report

A new 28-page research report by George Campbell's Security Executive Council (SEC) concerns the status of physical security metrics. Enterprise Security Metrics: A Snapshot Assessment of Practices (free but registration required) "provides a snapshot of the use of metrics in corporate security management. It includes information on the current state-of-the-art of various models of benchmarking and security metrics, types of metrics, judging the maturity of security metrics programs as well as challenges and opportunities for those undertaking security metrics programs. This report specifically summarizes our learned experience from corporate security measures and metrics initiatives."

The report refers to SEC's ongoing metrics research but unfortunately does not go into details about the methods.  A note on page 7 refers to a survey of 27 companies representing "a solid cross section of industry sectors [with] mature and multi-service corporate security programs, several engaging in best practice operations". The small sample was presumably drawn from members or clients of the SEC meaning that it was not random but self-selected from organizations with a clear interest in security metrics. Nevertheless, statistics aside, the findings and conclusions are well worth reading in more general terms - for example:

"Nearly 70 percent of respondents stated that they don’t collect security program metrics for the purposes of presenting to senior management ... This lack of engagement remains as a significant internal obstacle to metrics acceptance and development. Too many corporate security practitioners have either avoided or failed to understand the relevance of such measures. Security organizations have the data; they are willing to count events and other activity data but they apparently don’t see the need to use it to build actionable, influential metrics that can effectively influence senior management."

I like the phrase 'actionable, influential metrics'. Metrics that are neither actionable nor influential have little practical value. They are "coffee table metrics", the sorts of things one might idly skim through in a glossy magazine.  Metrics that are influential but not actionable can cause consternation: we know there is something wrong but we don't know how to fix it. Metrics that are actionable but not influential have no impact. Metrics that don't influence or support decisions are essentially pointless. Fow what it's worth, most such metrics tend to be cheap and easy to gather so the measurement costs are quite low, although there is a hidden cost in that they can be distracting, giving the impression that someone is on top of security metrics whereas in fact they are not.

The report mentions the commonplace KRIs and KPIs (Key Risk and Performance Indicators) plus two metrics that were new to me: Key Influence Indicators ("How do our metrics influence governance policy, business unit accountability and personal behavior?") and Key Value Indicators ("How have our metrics demonstrated tangible, actionable and measurable benefit to the enterprise?"). Influence and value are two of several characteristics of metrics, or metametrics. The PRAGMATIC method uses nine specific metametrics to determine the net value of a metric.

There is a common theme underlying the report's conclusions, namely that more effort should be put into identifying baseline metrics for all aspects of security in order to enable benchmarking comparisons between organizations. Security management practices and metrics requirements vary widely largely in practice because security risks vary widely, hence the particular security concerns that drive a given organization to select specific security metrics may not coincide with other organizations. However, an appendix to the SEC report offers a maturity metric measuring the status of an organization's [physical] security metrics program by assessing the anticipated parts of such a program. The metric is similar in style to those we described in PRAGMATIC Security Metrics, a form of metric that encourages us to break down and systematically assess complex situations within the organization (I found them well suited for internal audits and process improvement initiatives). Maturity metrics are also a promising approach for benchmarking comparisons of multiple organizations.

Another conclusion of the report is that metrics are needed for compliance assessment purposes: we discussed this point too in PRAGMATIC Security Metrics. Industry regulators and authorities (such as the other SEC!) need rational ways in which to measure and assess organizations on all sorts of criteria including governance, risk and security practices. The conventional approach is to specify and mandate certain requirements, in which case the measurement process boils down to someone (hopefully, a competent, independent and diligent third party) determining whether the stated requirements have or have not been fulfilled - fine in theory but harder to achieve in practice since there are so many variables. PCI-DSS, for instance, requires a number of specific security controls supposedly to secure cardholder data, and PCI assessments attempt to confirm that they are all in place. We know from Target and many other breaches that the PCI controls are imperfect, and that a "pass" on the PCI assessment does not necessarily mean that card holder data are in fact adequately secured. Furthermore, card holder data are just a fraction of most organizations' information assets, hence ticking the PCI compliance box does not necessarily mean the organization has adequate information security as a whole (it is an indicator, perhaps, but primarily concerns compliance with externally-imposed obligations). It would be practically impossible to extend the PCI-type approach to cover all of information security, physical security, risk management and governance, whereas other approaches such as maturity metrics could be used both to measure and to drive improvements.

George ends the report with a plea to collaborate with other metrics professionals. I welcome the initiative and will definitely get in touch! 

What are KPIs?

Krag and I have discussed this question from time to time and, although we are broadly aligned in our thinking, we haven't yet totally resolved our differences ... which makes the exchanges fun.

With that in mind, I always wonder what someone really means when they talk about KPIs. To some, Key Performance Indicator has a very specific and particular meaning, although I suspect if we assembled a dozen such people in a room to discuss it, we'd soon end up realizing that we have more than a dozen different interpretations! 

To others (including me, as it happens), KPI is a generic, blanket term for a class or type of metric that satisfies the criteria implied by the term: 

• Key implies that the metric itself is especially important, crucial or vital even, given that there are many many different ways to measure and assess things but most of them are of limited value. Picking out the few things that truly matter is a core issue in metrics. 'Spam volume' is an example of a metric that is both narrow and shallow, whereas 'email risk level' is a much broader, deeper and richer metric, and is far more likely to be considered key (even if we happen to be talking specifically about metrics relating to spam filtering). However, the criticality and value of metrics does depend on the contexts or situations being measured and the perspectives and information needs of their users. It is conceivable that 'spam volume' may be considered a KPI for the anti-spam controls, but that's a narrow perspective. Key may also refer to the performance, in the sense that the KPI is an indicator concerning an important issue; 
• Performance is a distinctly ambiguous word, implying concern for the process and/or its outcome. Are we measuring key activities (typically in order to assess and improve the efficiency of the process) or its outcomes (typically to assess and improve the effectiveness of the process), neither or both? I have seen KPI used in several different senses, although usually it is not totally clear (perhaps not even to the person discussing it!) which one is meant; 
• Indicator generally means simply a metric or measure but it may imply imprecision or approximation. A typical car's fuel gage, for instance, gives a fairly vague indication of the amount of fuel in the tank, whereas the equivalent metric on an aircraft tends to be much more precise, accurate and reliable, for obvious reasons. The car's fuel gage may not tell you how many litres remain but if it heads into the red zone, you know you need to find a filling station soon. Indicator often also implies a forward-looking or predictive rather than purely historical measure. 'Trends' are common examples, used to manage various aspects of information security where precision is nice but not vital (e.g. supporting security investment or resourcing decisions). 

As far as I'm concerned, then, a KPI is generally a predictive metric concerning some critical outcome of an important process. In the information security context, KPIs are most likely to measure core security processes such as risk assessment, and key controls such as authentication and access control. Efficiency is secondary to effectiveness, in that security failures resulting from ineffective controls can lead to serious and potentially very damaging incidents, whereas inefficient controls are merely a bit wasteful (that's actually a strong bias, one worth challenging in situations where security becomes onerous for information users and administrators, perhaps so onerous that they bypass or disable the controls sending us back to square one!).

Avoiding metrics myopia

Things being measured are patently being observed in a very specific, focused manner. Things that are observed so closely tend to be presumed by others to be of concern to the measurer/observer, at least. Things under the measurement ruler therefore assume a certain significance simply by dint of their being closely observed, regardless of any other factors.

We see this 'fascination bias' being actively exploited by cynical promoters, marketers and advertisers on a daily basis through an endless stream of largely banal and unscientific online polls and infographics, their true purpose made all the more obvious by the use of bright primary-color eye-catching graphics. They are manipulating readers to believe that since something has been measured, it must be important. How many of us take the trouble to think about the quality of the metrics, or about all the other aspects that haven't been measured? Like bunnies in the headlights, we stare at the numbers.

In PRAGMATIC Security Metrics, we outlined 21 observer biases drawn from an even longer list drawn up by Kahneman, Slovic and Tversky in Judgment Under Uncertainty (1982): what I'm calling 'fascination bias' has some resemblance to what Kahneman et al. described as 'attentional bias', the tendency to neglect relevant data when making judgments of a correlation or association.

Fascination bias creates a genuine concern in that we tend to measure things that are relatively easy to measure, and place undue faith in those metrics relative to other factors that are not being measured. Back in 2011, Michel Zalewski said in his blog:

"Using metrics as long-term performance indicators is a very dangerous path: they do not really tell you how secure you are, because we have absolutely no clue how to compute that. Instead, by focusing on hundreds of trivial and often irrelevant data points, they take your eyes off the new and the unknown."

While we don't entirely accept that we 'have no clue how to compute security performance', his point about neglecting other risks and challenges due to being inordinately focused on specific metrics is sound.  It's only natural that what gets measured gets addressed (though not necessarily improved!). The unfortunate corollary is that what doesn't get measured gets neglected.

The upshot of this is that there is a subtle obligation on those who choose metrics to find ways to measure all the important matters, even if some of those metrics are expensive/complex/qualitative/whatever. It's simply not good enough to measure the easy stuff, such as the numbers that assorted security systems constantly pump out 'for free'. It's inappropriate to disregard harder-to-measure issues such as culture, ethics, awareness and trust, just as it is inappropriate to restrict your metrics to IT or cybersecurity rather than information security.

That's one of the key reasons why we favor the systematic top-down GQM approach: if you start by figuring out the Goals or objectives of security, expand on the obvious Questions that arise and only then pick out a bunch of potential Metrics to answer those questions, it's much harder to overlook important factors. As to figuring out the goals or objectives, conceptual frameworks for information security such as BMIS and ISO27k, based on fundamental principles, are an obvious way to kick-off the thinking process and frame the initial discussion.

ISO27k Toolkit

On the toolkit theme, I have just updated the FREE ISO27k Toolkit over at ISO27001security.com with an Excel workbook used to track progress on implementing the ISO/IEC 27001 and 27002 standards.

Thanks mostly to Ed Hodgson, the gap analysis/SoA workbook in the ISO27k Toolkit has been updated for the 2013 releases of the standards.

The new version has two main spreadsheets:

1. The first sheet is used to check and track progress towards implementing an ISMS complying with all the mandatory front parts of ISO/IEC 27001:2013 - mandatory, that is, if you intend to get your ISMS certified.  I made a few little wording changes and editorial decisions in this section, so if you use this for certification purposes, please double-check against the requirements formally specified in the standard and don't rely entirely on the spreadsheet!  The spreadsheet is not definitive.  The standard rules.
   
2. The second sheet covers the discretionary parts, namely the controls listed briefly in Annex A of '27001 and explained in more depth in ISO/IEC 27002:2013 plus any controls that you add or change on the list, for example additional legal, regulatory or contractual obligations, or ISO 22301, NIST SP800s or whatever.  Don't be afraid to adapt the list of controls!  '27001 Annex A and '27002 are intended to be 'reasonably comprehensive' starting points, laying out a decent set of good security practices, but your information security risks and hence control requirements are unique to you. 

For both parts, you simply select the relevant colour-coded status indicator from a drop-down list on each item, and record brief notes to explain the situation.  The status levels are adapted from Carnegie Mellon's Capability Maturity Model, showing progress from not-implemented-at-all (bright red) up to fully-implemented-working-and-auditable (dark green), plus grey options for "? unknown" (i.e. status not yet checked) and "Not applicable".

A third metrics spreadsheet simply counts the number of items at each status level in each of the two main sheets, and draws a pair of pretty pie charts showing the proportions ... 

These very simple metrics clearly indicate progress towards a compliant, working, provable ISMS managing a reasonably comprehensive suite of information security controls, as both pies gradually go dark green.  [Pies going green is usually a bad sign, but in this particular case dark green pies are tasty  :-)]   Actually, in Excel, it is easy to generate whatever format charts you like, line graphs showing the trends month-by-month for instance. That's left as an exercise for the reader.

As with all the ISO27k Toolkit items, it is provided free under a Creative Commons license that allows you to use and adapt it as much as you want for your own purposes, and to share it under the same terms (but not to sell it!). 

I take full responsibility for any errors in the spreadsheets.  If/when you find any errors, please let me know.  My Excel skills are rudimentary: I'm sure an Excel wizard would be able to come up with a sexier version, with better error-checking, better usability, more easily extendable, and perhaps more functional.  If you do improve it, please share it back for the benefit of the whole user community.

PRAGMATIC security metrics toolkit

Krag and I have been thinking about what might be of value in a 'toolkit' for security metrics.  The kinds of things we have in mind are:

• Resources such as books and standards on information security, risk management, governance, metrics, statistics and business management - the toolkit would contain references, reviews and links, not the actual content!
• Techniques, methods and approaches - naturally I'm thinking of the PRAGMATIC method but there are alternative approaches (such as GQM) that complement it: again, the toolkit would contain just a summary with pointers to further advice, since there is a lot to be said;
• Bootstrap metrics - perhaps a few suggested information security metrics to get you started?  I'm not so sure about this because it's hard to think of information security measurement requirements that are widely applicable, but I guess we could come up with a few illustrative metrics ideas.  Oh wait, we did that already - the 150 metrics in the book, most of which have been discussed on this very blog!
• FAQ - we have made a start on a security metrics FAQ but there's plenty more scope to develop that.  What questions would you like us to address?
• Do's and don'ts - lessons from the trenches.  We particularly favor the idea of publishing case studies on organizations that have developed coherent suites of information security metrics, discussing the process, pitfalls, triumphs and outcomes.  Would your organization be a guinea pig?  If so, get in touch.
• A project plan - perhaps some sort of generic outline or template of the steps most organizations would anticipate taking to specify, design, develop and implement an information security measurement system?  

So what do you think?  Is a "PRAGMATIC security metrics toolkit" something that would interest you?  What would you most like it to contain?  What else, apart from the items listed above, would be most useful for you?  

Go ahead, we're all ears.

Contextually relevant information security metrics

In "Business Analytics - An Introduction", Evan Stubbs describes "value architecture" in these terms: "Results need to be measurable, they need to be contextually relevant, they need to link into a strategic vision, and their successful completion needs to be demonstrable".

Breaking that down, I find that there are really only two key factors. If results are measurable, that implies to me that they can be demonstrated. Also, it's hard to see how results that are 'contextually relevant' might not 'link into a strategic vision' since that is the context, or at least a major part of it. So, in short, results need to be both relevant and measurable.

Of those two aspects, measurability is the easier. Read "How to Measure Anything" by Douglas Hubbard! Evan also talks about objectivity, and he is writing in the context of big data analytics, meaning the difficult problem of extracting useful meaning from huge and dynamic volumes of complex data. Measurability is largely a matter of mathematics, or more precisely statistics. I agree it is a major issue but, with all due respect to the statistical wizzards, fairly mechanistic and logical.

That leaves the more awkward question of relevance. What are 'contextually relevant results'? Evan pointed out the strategic element, implying that relevant results can be extrapolated from corporate strategies. Strategies typically elaborate on how the organization intends to achieve identified long-term goals and interim objectives, often using the metaphor of a journey across a landscape passing waypoints en route to a destination. That in turn suggests the idea of measuring the actual direction and speed of travel - the trajectory - relative to the planned route as well as proximity to the eventual goal. Metaphorically speaking, it's more efficient to take a direct route than to be constantly side-tracked and diverted, and perhaps get lost.

So how does this relate to information security metrics? Evan implies that we need to defined the intended results of information security in terms that are both relevant and measurable.

Again I will side-step the measurability angle in order to focus on relevance. How is information security relevant to the organization?

In strategic terms, information security can be expressed in several different ways. Usually, we talk about protecting information assets, the defensive perspective. In this frame of reference, information systems, networks and information need to be defended against all manner of threats that would harm them. Relevant metrics here tend to relate to measuring and assessing the risks (threats, vulnerabilities and impacts, including security incident and business continuity metrics) and security controls (especially control efficiency and cost-effectiveness, implying most financial security metrics). Compliance is a classic defensive objective, hence compliance-related metrics also fit into this group.

Some of us also talk in terms of information security as a business enabler - letting the organization do business, safely, that would otherwise be too risky. Here we're thinking more proactively: security has an offensive as well as a defensive strategic role. Relevant metrics in this domain include the assurance angle, giving management confidence in the security arrangements so that they can concentrate on taking the most direct route. Hence control reliability metrics, plus various test and audit results, are in this group. Proactively exploiting strengths in information security also implies going beyond mere compliance (which, it has to be said, is a low hurdle) towards good or even best practice. Security maturity, benchmarking and governance metrics are relevant to business enablement. Measures of the integration of information security into various business and IT processes and systems are an example.

What are we left with?  Mmm, I'm not sure I can think of any information security metric that doesn't fit into one or other of those categories. Can you?

Holistic security metrics

Yet again today I find my blood pressure reading as I read yet another incredibly biased pronouncement on security metrics from security vendors:

"Do you know what security metrics are right for your organization? For a holistic view, both network and host metrics are required, including firewalls, routers, load balancers, and hosts."

To claim that having network and host security metrics qualifies as holistic almost beggars belief, for any thinking person's definition of the term but I'm afraid it's typical of the incredibly myopic purely technical perspective on security metrics, continually reiterated for blatantly obvious marketing reasons by the purveyors of ... IT security products.

Being sick and tired of explaining that IT security is a dead end off the main information security highway, I'll merely suggest a few non-technical security metrics that might get us a tiny bit closer towards a truly holistic view:

• Information security ascendancy - a measure of the importance of information security in the eyes of management
• Value of information assets owned by each information asset owner - brings up the whole notion of accountability and responsibility for protecting valuable and yet vulnerable information
• Security governance maturity - mmm, governance, yes, I've heard of that
• Business continuity plan maintenance status - oh I suppose we ought to consider the remote possibility that our expensive antivirus systems and firewalls might not be the perfect solutions we were sold
• Corporation's economic situation - can we even afford those fancy security gadgets?
• Number of important operations with documented & tested security procedures - oh yes, procedural controls, I've heard of them
• Information security budget variance - a lone example of a substantial class of financial metrics that are distinctly relevant to both information security and the business
• Security metametrics, such as PRAGMATIC - a way to sift seeds from chaff.

For an holistic view of information security, I respectfully submit that "network and host metrics" fall woefully short of sufficient.  They are needed, yes, but they are definitely not enough.