CIS cyber security metrics

The latest and greatest sixth version of the CIS (Center for Internet Security) Critical Security Controls (now dubbed the "CIS Controls For Effective Cyber Defense") is supported by a companion guide to the associated metrics. Something shiny in the introduction to the guide caught my beady eye:

"There are lots of things that can be measured, but it is very unclear which of them are in fact worth measuring (in terms of adding value to security decisions)."

Sounds familiar. In PRAGMATIC Security Metrics, we said:

"There is no shortage of ‘things that could be measured’ in relation to information security. Anything that changes can be measured both in terms of the amount and the rate of observable change, and possibly in other dimensions as well. Given the dynamic and complex nature of information security, there are a great number of things we could measure. It’s really not hard to come up with a long list of potential security metrics, all candidates for our information security measurement system. For our purposes, the trick will be to find those things that both (a) relate in a reasonably consistent manner to information security, preferably in a forward-looking manner, and (b) are relevant to someone in the course of doing their job, in other words they have purpose and utility for security management."

From there on, though, we part company. 

The CIS approach is highly prescriptive. They have explicitly identified and detailed very specific metrics for each of the recommended controls. For example, the metric associated with control 4.5:

"Deploy automated patch management tools and software update tools for operating system and software/applications on all systems for which such tools are available and safe. Patches should be applied to all systems, even systems that are properly air gapped."

asks 

"How long does it take, on average, to completely deploy application software updates to a business system (by business unit)?". 

To answer that particular question, three distinct values are suggested, viz 1,440, 10,080 or 43,200 minutes (that's a day, a week or a month in old money). It is implied that those are categories or rough guides for the response, so why on Earth they felt the need to specify such precise numbers is beyond me. Curiously, precisely the same three values are used in most if not all of the other suggested metrics relating to time periods ... which might be convenient but disregards the differing priorities/timescales likely in practice. I'd have thought some controls are rather more urgent than others. For instance, the time needed by the organization to restore normal IT services following a disaster is markedly different to that required by an intrusion detection system to respond to a identified intrusion attempt. These are not even in the same ballpark.

The same concern applies to the CIS' proportional metrics. The suggested three choices in all cases are "Less than 1%", "1% to 4%" or "5% to 10%".  

Note that for both types, answers above the maximum value are unspecified.

Note also that the response categories cover different ranges for those types of metric. The timescale values are roughly exponential or logarithmic, whereas the proportions are more linear ... but just as arbitrary. 

Oh and the timescales are point values, whereas the proportions are ranges.

The only rationale presented in the paper for the values is this vagueness:

"For each Measure, we present Metrics, which consist of three “Risk Threshold” values. These values represent an opinion from experienced practitioners, and are not derived from any specific empirical data set or analytic model. These are offered as a way for adopters of the Controls to think about and choose Metrics in the context of their own security improvement programs."

Aside from the curious distinction between measures and metrics, what are we to understand by 'risk thresholds'? Who knows? They are hinting at readers adapting or customizing the values (if not the metrics) but I rather suspect that those who most value the CIS advice would simply accept their suggestions as-is.

Later in the metrics paper, the style of metrics changes to this:

"CSC 1: Inventory of Authorized and Unauthorized Devices - Effectiveness Test. To evaluate the implementation of CSC 1 on a periodic basis, the evaluation team will connect hardened test systems to at least 10 locations on the network, including a selection of subnets associated with demilitarized zones (DMZs), workstations, and servers. Two of the systems must be included in the asset inventory database, while the other systems are not. The evaluation team must then verify that the systems generate an alert or email notice regarding the newly connected systems within 24 hours of the test machines being connected to the network. The evaluation team must verify that the system provides details of the location of all the test machines connected to the network. For those test machines included in the asset inventory, the team must also verify that the system provides information about the asset owner."

As I said, this is a highly prescriptive approach, very specific and detailed on the measurement method. It's the kind of thing that might be appropriate for formalized situations where some authority directs a bunch of subserviant organizations, business units, sites or whatever to generate data in a standardized manner, allowing direct, valid comparisons between them all (assuming they follow the instructions precisely, which further implies the need for compliance activities).

Anyway, despite my criticisms, I recommend checking out the CIS critical controls for cyber defense. Well worth contemplating.

Zurich Insurance global cyber risk reports

Zurich Insurance published a web page with a bunch of graphs projecting the global costs and benefits of cybersecurity under various scenarios ... but what do they mean? What is the basis for analysis? I find the graphs confusing, almost devoid of meaning like so many infographics, a triumph of marketing gloss over substance. The page succeeded, however, in catching my beady eye.

Although Zurich neglected to provide a working hyperlink, Google led me inexorably to the research paper from which the graphs were plucked: Risk Nexus: Overcome by Cyber Risks? Economic Benefits and Costs of Alternate Cyber Futures is a report by the Zurich Insurance Group and the Atlantic Council's Brent Scowcroft Center on International Security plus the Pardee Center for International Futures at the University of Denver, a follow-up to their 2014 report: Beyond Data Breaches: Global Aggregations of Cyber Risk.   

Apart from casually referring to "cyberspace" as 'the internet and associated IT', the reports are littered with undefined/vague cyber terms such as "cyber risks", "cyber attacks", "cyber crime", "cyber incidents", "cyber shocks" and "cyber futures". You might be comfortable with "cyber" but replacing it with "Internet-related" suits me better since they are not talking about information or IT security in general, nor about cyberwar in particular - two other common cyber-interpretations.

The 2014 report

The 2014 report conjured up and considered a potential disaster scenario involving a major Internet-related incident at a large communications technology firm triggering cascading failures affecting the global economy, in other words a systemic risk with global repurcussions:

"Early on, we nicknamed this project ‘cyber sub-prime’ because we intended it to expose the global aggregations of cyber risk as analogous to those risks that were overlooked in the U.S. sub-prime mortgage market. Problems in that segment spread far beyond the institutions that took the original risks, and proved severe enough to administer a shock that reverberated throughout the entire global economy. At first, the term ‘cyber sub-prime’ was just a quirky nickname, but it soon became a useful analogy, helping us to gain additional insights into cyber risks based on extended parallels with the financial sector."

While there is value in drawing lessons from the global financial crisis, I wonder if maybe the research team has been blinkered into that particular mode of thinking or world view, ignoring other possible futures such as, say, terrorism or more gradual as opposed to sudden crises, overpopulation for example? 

Anyway, the report recommended "several concrete steps that must be taken to overcome these inevitable shocks of the future and prevent what could be called a 'cyber sub-prime' meltdown.  Recommendations to be resilient to cyber shocks include:

• Putting the private sector at the center of crisis management, since government management of cyber risk lacks the agility needed

• Developing plans within organizations that have system-wide responsibility that ensure the stability of the system as a whole, rather than risks to an individual organization

• Creating redundant power and telecommunications suppliers and alternate ISPs connect to different peering points

• Investing in trained teams ready to respond with defined procedures

• Conducting simulations of the most likely and most dangerous cyber risks to better prepare"

I appreciate what they are getting at in the first bullet but I'm not sure I agree with it. The private sector may arguably be more 'agile' in managing Internet-related risks, but overall is it doing any better in fact? I see little evidence that the private sector is any more highly protected than the government sector, particularly given differences in the nature of their respective risks. Even if that's true, why did they ignore or discount the obvious strategic option of improving government sector Internet-related security, I wonder? Perhaps the fact that the research was funded by a private-sector insurance company has something to do with it ... 

Their other points about considering systemic risk and developing more resilient infrastructures, effective incident response and training exercises involving simulations are fine by me, conventional and widely supported. The possibility of complete, permanent failure of the Internet is but one of several extreme disaster scenarios that I recommend clients consider for information risk and business continuity management purposes. My key point is not to plan too narrowly for any one particular scenario (or in fact any of the unbounded set of credible situations that could lead to such an outcome, such as an all-out cyberwar) but to use a wide variety of diverse scenarios to develop more comprehensive resilience, recovery and contingency arrangements in a far more general sense. Preparing for the worst case has benefits under less extreme conditions too, while there are far too many scary possibilities to risk being unprepared for what actually transpires.

As to whether those five bullets constitute "concrete steps", I guess it's a matter of perspective or terminology. The report stops well short of providing pragmatic action plans and allocating responsibilities. Not so much rock-hard concrete as sloppy mud! [In contrast, take a look at the ICAO Global Aviation Safety Plan, a strategic approach to ensure continued safety in the global aviation industry, laying out specific actions, responsibilities and timescales: now that's what I call concrete!]

The 2015 report

The risk and economic modeling study evidently continued, leading to last year's report.  I'll leave you to cast a cynical eye over the latest report. I'm too jaded to take it seriously.

Security awareness metrics

Some say that information security awareness is hard to measure, and yet a moment's thought reveals several obvious, straightforward and commonplace metrics in this area, such as:

• Attendance numbers, trends, rates or proportions at awareness and training events;
• Feedback scores and comments from attendees at/participants in said events, or concerning other awareness activities, promotions, media, messages etc.; 
• General, broad-brush, state-of-the-nation security awareness surveys of various populations or constituencies conducted on paper or using electronic forms or polls;
• More specific information recall and comprehension tests relating to awareness topics or sessions, conducted on paper or online (maybe through the Learning Management System);
• Awareness program metrics concerning activities planned and completed, topics covered (breadth and depth of coverage), budget and expenditure ($ and man-days), comparisons against other forms of security control and against other awareness programs (in other fields and/or other organizations). 

With a little more thinking time, it's quite easy (for me, anyway) to come up with a broader selection of awareness metrics also worth considering: 

• More elaborate versions of the above, perhaps combining metrics for more meaningful analysis - for instance using attendance records and feedback to compare the popularity and effectiveness of different types of awareness and training events, different topics, different timings, different presenters, different media etc.;
• Page hit rates, stickiness and various other webserver metrics concerning the popularity of/interest in the information security intranet site, including various elements within it, such as the security policies and specific topic areas;
• Metrics gleaned from personnel records (e.g. proportions of the workforce with basic, intermediate or advanced qualifications, or with skills and competencies relating to information security, privacy, governance, risk etc., and currency of their skills, knowledge, competencies and qualifications);
• Targeted surveys/polls comparing and contrasting awareness levels between various groups (e.g. different business units, departments, teams, levels, specialisms, ages, sexes, cultures/nationalities etc.) or times (e.g. before, during and after specific awareness/training events, awareness focus periods, business periods etc.) or topics (e.g. phishing vs. other forms of social engineering, malware, fraud etc.);
• Workforce security awareness/culture surveys and studies conducted in person by trained and competent survey/research teams (a more expensive method that can generate better quality, richer, more valuable information);
• Maturity metrics using audits, reviews, surveys and self-assessments to determine the maturity and quality of the organization's overall approach to security awareness and training relative to the state of the art in awareness (as documented in various standards, books and websites);
• Benchmarking - comparing information security awareness levels, activities, spending etc. against other fields (such as health and safety or legal compliance) or organizations, industries etc.;
• Risk-based awareness metrics, perhaps assessing the relevance of employee awareness, understanding, knowledge, competence, responsiveness etc. to various information risks, issues or challenges facing the organization, giving a natural priority to the planned awareness and training topics and a basis for budgeting (including resourcing for the security awareness and training program);
• Risk-based information security metrics looking at myriad sources to identify current information risks, trends, predictions, technology directions, emerging threats etc. (useful for strategic planning in information security, of course, with an obvious link through to the corresponding awareness and training needs);
• Change metrics concerning change management and changes affecting the organization, especially those relevant to information risk, security, privacy etc., as well as measuring and driving changes within the awareness program itself;
• Process metrics concerning various information risk, security, privacy, governance and compliance-related processes (again including those concerning awareness and training) and various parameters thereof (e.g. cost and effort, efficiency, effectiveness, consistency, complexity, compliance, creativity, risk ...); 
• Quality metrics concerning the awareness content/materials including policies, procedures and guidelines: there are many possible parameters here e.g. the style of writing and graphics, professionalism, review and authorization status, breadth and depth of coverage, currency/topicality and relevance, readability (e.g. Flesch scores), interest/engagement levels, consistency;
• Awareness surveys conducted by information security presenters, trainers and other professionals: people attending training courses, conferences, workshops and so forth are generally accustomed to completing survey/feedback forms concerning the events e.g. the quality and competence of the presenter/trainer/facilitator, the materials, the venue, the catering etc. and, fair enough, that's quite useful information for the planners of such events. Why not also get the people who present/train/facilitate/lead the events to rate their audiences as well, on parameters such as interest in the topic, engagement, knowledge levels, receptiveness etc.?  Your Information Security Management, Security Admin, Help Desk, PC Support, Risk and Compliance people will have a pretty good idea about awareness and competence levels around the organization. Management, as a whole, knows this stuff too, and so do the auditors ... so ask them!;
• Customer contact metrics for the information security team including the security awareness people, measuring the nature and extent of their interactions with people both within and without the business (e.g. their attendance at professional meetings, conferences, webinars, courses etc.);
• Various awareness metrics gleaned from Help Desk/incident records relating to events and incidents reported (e.g. mean time to report, as well as mean time to resolve, incidents), help requests (number and complexity, perhaps split out by business unit or department), issues known or believed to have been caused by ignorance/carelessness etc., as well as general security metrics concerning incident rates for various types of information security incident - another driver to prioritize the planning and coverage of your awareness activities.

I could continue but even my eyes are glazing over at this point, so instead I want to end with some quick comments about how to make sense of all those and other options, and how you might go about selecting 'a few good security awareness metrics' that might be worth actually using.

Two specific approaches I recommend are PRAGMATIC and GQM.  

GQM starts with some exploration and analysis of your organization's goals or strategic objectives for information risk, security, privacy, governance, compliance and all that jazz (especially how these aspects support or enable core business), leading to some fairly obvious high-level questions (e.g. "Are we sufficiently compliant with our legal obligations towards privacy?") and thence to the kinds of metrics that would generate the data that might address or answer those questions (privacy compliance metrics in that case).   At a lower level of detail, the same approach can be used to determine the goals, questions and kinds of metrics for security awareness.  [Sorry, I'm not going to do that for you - it's your homework for today!]  [For more on GQM, read Lance Hayden's book IT Security Metrics].

PRAGMATIC is a rational basis for choosing between a bunch of possible metrics and assorted variants, or to guide the creative development of new metrics, or to drive improvement by weeding out ineffective metrics and getting more value out of those that remain, using nine key criteria or parameters for metrics: Predictiveness, Relevance, Actionability, Genuineness, Meaninfulness, Accuracy, Timeliness, Integrity/Independence and Cost-effectiveness.  [For more on PRAGMATIC, read our book PRAGMATIC Security Metrics, browse this website or blog, or ask me!]

We don't know, we just don't know UPDATED

Crime-related metrics are troublesome for several reasons.  

Firstly, crime tends to be hidden, out of sight, mostly in the shadows. An unknown number of crimes are never discovered, hence recognized/identified incidents may not be representative of the entire population. Criminals might brag about their exploits to their posse but they are hardly likely to participate willingly in surveys.

Secondly, criminals can't be trusted so even if they did complete the forms, we probably shouldn't swallow their responses. Mind you, if the surveys weren't designed scientifically with extreme care over the precise questions, proper selection of the samples, rigorous statistical analysis, honest reporting etc., then all bets are off. 

Thirdly, the police, governments/authorities, the news media, assorted commercial organizations, professions, industry bodies and pressure groups all have vested interests too, meaning that we probably shouldn't believe their surveys and assessments either, at least not uncritically*. Guess what, if an organization's income or power depends to some extent on the size of The Problem, they may, conceivably, allegedly, be tempted to slightly over-emphasize things, perhaps exaggerating, oh just a little and down-playing or ignoring inconvenient metrics and findings that don't quite align with their world view or objectives. [This one applies to me too as an infosec pro, but recognizing my inherent bias is not the same as counteracting it.]

Fourthly, the metrics vary, for example in how they define or categorize crimes, what countries or areas they cover, and the measurement methods employed. Are US homicide numbers directly comparable with murders in, say, the UK? Are they even comparable, period-on-period, within any constituency? Would deliberately killing someone by running them over 'count' as a car crime, murder, accident, crime of passion, and/or what?

Fifthly, the effects of crime are also hard to account for, especially if you appreciate that they extend beyond the immediate victims. Society as a whole suffers in all sorts of ways because of crime. These effects and the associated costs are widely distributed. 

Sixthly, and lastly for now, crime is inherently scary, hence crime metrics are scary or eye-catching anyway. We risk losing our sense of perspective when considering 'facts' such as the skyrocketing rates of gun crime, home invasions, child abductions or whatever in relation to all the normal humdrum risks of everyday life, let alone all those scares about smoking, obesity, stress, heart disease and cancer. The emotional impact of crime metrics and the way they are portrayed in various media introduces yet more bias. [By the way, the same consideration applies to security metrics: perhaps we should explore that tangent another day.]

So, with all that and more in mind, what are we to make of cybercrime? How many cybercrimes are there? How many remain unidentified? To what extent can we trust our information sources? How do we even define, let alone measure, cybercrime? What is The Problem, and how big is it? And does it really matter anyway if the answer is bound to be scary?

Well yes it does matter because all sorts of things are predicated on cybercrime statistics - strategies, policies (public, corporate and personal), risk assessments, investment and spending plans, budgets and so forth. 

The right answer might be: we don't know. Good luck with all those predicates if that's your final answer! Phone a friend? 50/50?

* Update Feb 20th: according to Cybercrime costs more than you think, "Cybercrime costs the global economy about $450 billion each year", a factoid used (for reasons that are not entirely obvious) to support a call for organizations to plan for incidents. Their sources are not clearly referenced but the paper appears to draw on a glossy report by Allianz, an insurance company with an obvious self-interest in pumping-up the threat level. The Allianz report in turn cited studies by the Ponemon Institute and by McAfee with the Center for Strategic and International Studies, three further organizations with axes to grind in this space. To their credit, the 2014 McAfee/CSIS study openly acknowledged the poor quality of the available data - for instance stating: "... we found two divergent estimates for the European Union, one saying losses in the EU totaled only $16 billion, far less than the aggregate for those EU countries where we could find data, and another putting losses for the EU at close to a trillion dollars, more than we could find for the entire world ..." They also noted particular difficulties in estimating the costs of theft of intellectual property, while simultaneously claiming that IP theft is the most significant component of loss. Naturally, such carefully-worded caveats buried deep in the guts of the McAfee/CSIS study didn't quite make it through to the Allianz glossy or the sales leaflets that cite it. It's a neat example of how, once you unpick things, you discover that incomplete and unreliable information, coupled with rumours, intuition, guesswork, marketing hyperbole and weasel words, have morphed via factoids, soundbytes and headline horrors into 'fact'. Hardly a sound basis for strategic decision-making, or indeed for purchasing commercial goods and services. 

Cause =/= Effect

Animals like us are fantastic at spotting patterns in things - it's an inherent part of our biology, involving parts of our brains that are especially good at it. Unfortunately, while some patterns are significant, many are not, and our brains are not terribly good at differentiating between the two - in fact, we tend to overemphasize matches, believing them to be especially significant, meaningful and, in a sense, real.

It could be argued that both pattern-recognition and overemphasis on matches are the result of natural selection over millenia, since in the wild, anything that helps us quickly identify and respond to possible attacks by predators, even if there are none, is likely to increase our survival, within reason anyway. Arguably, this is what makes wild animals 'alert', 'nervous' or 'jumpy'. It's a fail-safe mechanism. It's also the root of the fear we feel when we think we are in a dangerous situation, such as walking down a dark alleyway in an unfamiliar city at night. The sense of physical danger heightens our senses and primes our fight-or-flight instincts with a boost of adrenaline. Running away screaming from a harmless vagrant is safer than ignoring potential threats.

However, what I've just done in that paragraph is invent a vaguely plausible scenario, outlined it briefly, and some of you now believe it to be true, based on nothing more than its apparent plausibility and my credibility (such as it is). The reason I mentioned running away screaming was to stimulate a visceral reaction in you: the strong emotions that situation invokes adds even more emphasis to the story.  It 'makes sense'. In fact, there are many other plausible scenarios or reasons why pattern-recognition and overemphasis might or might not be linked to anything but having described a particular pattern, that is probably now locked into your brain and perhaps given special significance or meaning.

To illustrate my point, look at pattern-recognition from the predator's perspective: predators need to recognize possible prey and respond ahead of competing predators ... but distinguishing edible prey from everything else (including other predators, animals with poisonous or otherwise dangerous defenses, and rocks) is a critical part of the predator's biology. Attacking anything and everything would be a fail-unsafe approach, the exact opposite of prey. In reality, there are very few 'pure' predators or prey: even prey animals need to eat, while apex predators at the very top of the food chain may have a fear of cannibalism or prey that successfully fights back, so the real world is far more complex that my simplistic description implies.

OK, with that in mind, take a look at this graph:

Sure looks like the red and black lines are related, doesn't it? They track each other, on the whole. Their patterns match quite closely over the 13 year period shown, implying that they are somehow linked. In that specific case, statistical analysis tells us that the two variables are indeed correlated with a probability of just under 79% where 100% represent total identicality (indistinguishable) and 0% represents total discrepancy (no relation whatsoever). 79% is a pretty high value, so it is entirely possible that the two variables are indeed linked. 

So, at this point we think we've found a link between <ahem> the annual number of non-commercial space launches globally and the annual number of sociology doctorates awarded in the US - for those are the numbers graphed! Hmmmm.

Yes, you might be able to come up with some vaguely credible reasoning to explain that apparent linkage, but be honest it would be a stretch of the imagination and would involve considerable effort to find, which you might be willing to do if you feel the pattern-match is somehow significant (!). Far more likely is that we've simply found a matching pattern, a sheer coincidence, a fluke. If we have enough data available and keep on searching, we can probably find other variables that appear to correlate with either of those two, including some with even higher coefficients of correlation ...

... which I guess is pretty much what someone has done - using automated statistical techniques to find correlations between published data. Have a browse through these spurious correlations for some 29,999 other examples along these lines, and remember all this the next time you see a graph or a description that appears to indicate cause-and-effect linkages between anything. We humans desperately want to see matches. We find them almost irresistable and especially significant, almost magical, verging on real. Unfortunately, we are easily deluded.

From that point, it is but a short hop to 'lies, damn lies, and statistics'. Anyone with an axe to grind, sufficient data and a basic grasp of statistics can probably find correlations between things that appear to bolster their claims, and a substantial proportion of their audience will be swayed by it, hijacked by their own biology. I rather suspect that civil servants, politicians and managers are pretty good at that.

By the way, although I recognise the bias, I am far from immune to it. I try to hold back from claiming causal links purely on the basis of patterns in the numbers, and phrase things carefully to leave an element of doubt, but it's hard to fight against my own physiology.

Think on.

Gary.

PS. Finding spurious matches in large data sets is an illustration of the birthday paradox: there is a surprisingly high probability that two non-twin students in the average class were born on the same day. 

PPS The 79% correlation in the example above is only a fraction beneath the 'magical' 80% level. According to Pareto's Principle (I'm paraphrasing), 80% of stuff is caused by 20% of things. It's a rule-of-thumb that seems to apply in some cases, hence we subconsciouly believe it can be generalized, and before you know it, it's accepted as truth. The fact that 80% + 20% = 100% is somehow 'special' - it's another obvious but entirely spurious pattern.