Management without metrics - how?

The SEC (Security Executive Council - not the Securities and Exchange Commission!) boldly describes itself as "the leading research and advisory firm that specializes in security risk mitigation."  Their primary interest appears to be physical security, although they also make the odd nod towards IT security, business continuity and 'convergence'.

The SEC conducted an unscientific online poll, asking respondents to self-assess and report the capability maturity of their security programs using the classic 5 point SEI-CMM scale.  Unsurprisingly, the results show a vaguely normal distribution about the middle value ('defined'), skewed towards the low end of the maturity scale.

It appears they may have asked a separate question about metrics:

"When participants were asked about metrics (a higher level of maturity), 64% said they did not use business value metrics (metrics that are beyond initial "counting" of activities such as number of background checks performed or number of badges issued)."

So only about a third of their respondents have security metrics other than the absolute basics - a pathetically low proportion that begs the obvious question "How are they managing security without metrics?"

Answers on a postcard please.  Or comment below.

Metrics matter (updated)

An article by Mintz Levin about the 2013 privacy breach/information security incident at US retailer Target stated that the company has disclosed gross costs of $252 million, with some $90m recovered from its insurer leading to a net cost of $162m, up to the end of 2014 anyway (the incident is not over yet!).

Given that the breach apparently involved personal information on about 40 million people, it's trivial to work out that the incident apparently cost Target roughly $6 per compromised record ($4/record net of insurance payouts) ... but before anyone runs amock with those headline numbers, let's delve a bit deeper.

First off, what confidence do we have in the numbers themselves? The article cites its sources as 8-K filings, in other words Target's official reports concerning the incident to the Securities and Exchange Commission. Personally, I'm quite happy with that: the dollar amounts are not mere speculation but (I believe, not being a specialist in US laws and regulations) have been carefully drawn up, audited and formally approved by management - specifically Target's CFO. We could pore over the filed 8-K reports to verify them since they are published on the SEC site, or we could simply accept Mintz Levin's word: they are a law firm so there's a degree of trust and confidence. 

I took the 40 million compromised record count from a convenient web page somewhere - not as easy to verify but there are many such pages reporting similar numbers, so let's assume the figure is based on a count and disclosure by Target.  And let's assume it's correct (yes, another assumption).

Now dig further. Having tried to track and calculate the financial costs from relatively small information security incidents myself, I appreciate just how tough that can be in practice. The costs fall across two main categories, direct and indirect or consequential. The direct costs are a bit of a nightmare to monitor when everyone is running around frantically dealing with the incident at its height, but they can be estimated retrospectively and tracked fairly accurately once things calm down: it's a matter of cost accounting. Simply stated, someone assigns the direct expenses associated with the incident to an accounting code for the incident, and the financial system tots up and spews out the numbers. There are several opportunities for substantial error in there (for instance, signficant costs wrongly coded or neglected, and investments in information security/privacy improvements that would have been made anyway, regardless of the incident, being charged against it in order to secure the budgets and inflate the insurance claims), but these errors pale into insignificance against the indirect or consequential costs ...

A serious information security incident that becomes public knowledge seems likely to have an adverse impact on the organization's image and hence its brand values, but how much of an effect, in dollar terms? It's almost impossible to say with any certainty. In the case of a major incident, the company's marketing and financial people could evaluate and estimate the effects using metrics such as customer footfall, turnover, profitability, market surveys and so forth ... but potentially there is a conflict of interest there since those self-same people are charged with maintaining or boosting the company's brands and value, hence they may be understandably reluctant to report bad news to management. Furthermore, there are no easy, generally-accepted, accurate or independently-verifiable ways to convert changes in most of these metrics (such as "brand recognition") into dollars without a great deal of argument and doubt.

On top of that, there is some truth to the saying that "There's no such thing as bad news". Publicity of incidents is also publicity for the organizations and individuals involved. Publicity equates to media exposure and brand recognition hence, paradoxically, bad incidents might actually benefit those involved.

That leads us to consider stock price as another possible measure of the gross effects of an incident, one that conveniently enough is already in dollars and is widely reported, with historical data just a few click away (e.g. see the 3-year Target share price graph to the left here, courtesy of those nice people at MarketWatch.com). Given the number of shares issued (requiring a few more clicks), it's not too hard to convert the share price at any point into a market capitalization value for the company, and thus to calculate the effect the incident had on that value, but now it gets really interesting. After the incident was initially disclosed and widely reported in 2013, Target's share price declined markedly and then recovered in 2014, and is now well above the 2013 peak. What relation does that have to the incident? Again, it's almost impossible to say because there are just so many factors involved: stockbrokers, dealers and investors take a professional interest in identifying, evaluating and predicting those factors, and some of them are very successful so you might try asking them about the incident, but don't be surprised if they confuse you with statistics while keeping their trade secrets to themselves!

The same issue cropped up in the Sony hack at the end of last year. Sony's share price (plotted on the right over the past 6 months) has moved quite consistently upwards. There was a noticeable dip around the year end but it pretty much recovered its original trajectory by the end of January. I'm quite sure I could fit a straight line trend to the data with little statistical variance. 

OK, is that all there is to it? Well, no, we're not finished yet, not by a long chalk. 

So far we've only considered Target's costs: what about those whose personal information was disclosed, and the banks and other companies who have lost out to identity fraud? How much has the incident as a whole cost? How on Earth can we measure or calculate that? Once again, the short answer is that we can only estimate at best. 

What price would YOU put on the personal aggravation and grief caused by discovering that YOUR privacy has been breached and you may be the victim of identity theft? Go ahead, think about it and name your price! If enough of us did so, we might generate some sort of mean value but it's obviously highly subjective and doubtless extremely sensitive to the context and the precise questions we pose - plus of course there's the issue of our sampling strategy and sample size, since we can't ask everyone. Unfortunately, even a small error in our per-victim cost estimate will be massively amplified if we multiply that by the 40 million, so we really ought to take more care over this if the numbers matter - which they surely do as we'll come on to in a moment.

First, though, consider that the relationship between the total cost of a privacy breach/incident and the number of records disclosed is generally implied, but that is another unproven and potentially highly misleading assumption. We don't actually know the nature of the relationship, and it is likely to vary according to a number of factors aside from just the number of records. Identities belonging to the rich and famous are probably worth much more to identity thieves than those belonging to the the poor, for example, so a breach involving data from high-worth individuals, organizations or celebrities seems likely to result in greater losses than one involving the same number of records for "ordinary" people. Different items or types of information vary markedly in their inherent value (e.g. contrast the value of someone's email address or phone number to their credit card number - and then consider the additional value to fraudsters of obtaining multiple items in linked records). One might argue on basic arithmetic that the per-record costs decrease exponentially as the number of records increases, or that the relationship is non-linear due to the additional impact of news headlines with nice round figures ("more than 40 million" is worse than "almost 40 million", and far worse than "40 thousand"!). 

In privacy breaches, the black-market price of credit card numbers etc. is sometimes used to estimate the overall costs (e.g. if 'the average' record is worth, say, $2 to criminals, then $40m records are worth $80m). That simplistic approach begs various questions about how we determine the black-market price (which, by its very nature, is not openly available information), and at what point we measure it (since the value of stolen credit card numbers declines quite rapidly as word about the incident spreads and victims, banks and credit card companies progressively identify and cancel the cards). Furthermore, the costs accruing to the victims (i.e. Target and its owners/stakeholders, the data subjects, the banks and other institutions involved, oh and the FBI, police etc.) as a result of the incident may be related to but almost certainly exceed the profits accruing to the identity thieves, fraudsters and assorted middle-men exploiting it. Society as a whole picks up the discrepancies in a diffuse fashion.

That brings us to our final issue. Who cares how much infosec incidents such as this actually cost anyway? It matters because the information gets used in all sorts of ways, for example to justify investment in information security and privacy controls, incident management, insurance premiums, identity theft cover, contingency sums and more. It gets used for budgeting and benchmarking, for policy- and law-making. It feeds into our general appreciation of the information risks associated with personal information, and information risks as a whole. 

Stepping back a pace or two, this whole issue could be considered the elephant in the room for information risk and security professionals. We put enormous effort into promoting and justifying investments in information security controls to reduce the probability of, and damage caused by, incidents, trying our level best to persuade management to take heed of our concerns, support our business cases and invest adequately in security, especially proactive measures, systematic approaches and good practices such as ISO27k ... but if we look coldly and dispassionately at the situation including the assumptions and arguments laid out above, it could be said that incidents are not nearly as bad as we tend to make out, in other words we are crying wolf.  

Oh oh!  I guess we ought to firm up some of those estimates and assumptions, pronto, before we all lose our jobs! Metrics do matter, in fact.

PS The 2015 Verizon Data Breach Investigation Report attempts to define the mathematical relationship between 'Payout' and 'Records Lost' in so-called data breach incidents (see figure 21 and associated text), but acknowledges that although they have improved their model, they still don't have a firm grasp of all the relevant factors. Perhaps this blog piece will prompt them to re-evaluate their assumptions and presumptions, maybe even to do the research given the data and other resources available to them. Don't hold your breath though. I fully expect the mythical linkage between incident costs and records compromised to persist for many years yet, despite my best efforts. Its the infosec equivalent of the search for the holy grail - the Monty Python version. 

Comparative security metrics

In situations where it is infeasible or impracticable to quantify something in the form of a discrete count or a value in specific units, comparative or relative measures are a useful alternative. They are better than not measuring at all, and in some cases easier to comprehend and more useful in a practical sense. In this respect, we disagree with those in the field who fervently insist that all metrics must be expressed as numbers of units (e.g. "20 centimetres"). It seems to us "A bit longer than a pencil", while obviously imprecise, might be a perfectly legitimate and helpful measure of something (regardless of what that thing might be - a cut on your arm for instance).

Cardinal numbers and units of measure have their place, of course, but so do ordinals, comparatives and even highly subjective measures - all the way down to sheer guesswork (and, yes, 'down to' itself implies a comparative value). Douglas Hubbard's "How To Measure Anything" is an excellent, throught-provoking treatise on this very subject.

In information security, comparisons or relations can provide answers to entirely valid and worthwhile questions such as:

• Are we more or less secure than our peers?
• Are we getting more or less secure over time?
• If we both sustain our present rate of change, how long will it be before we'll surpass our competitors' level of information security?
• Are our information risks increasing or decreasing?
• Which are our strongest and weakest areas or aspects of security?
• Of all the myriad changes currently occuring in information security, what are the most worrying trends?
• Does information risk X fall within or exceed our risk appetite or tolerance?
• Which business unit, function, department or site is the most/least vulnerable?
• Are we spending too little, about the right amount, or too much on information security

As part of an information security awareness case study on 'the Sony hack', a management discussion paper describes three types of comparative security metrics with several examples of each.

Management awareness paper on contingency metrics

Here's the next security awareness paper in the series, describing metrics relating to contingency and business continuity management.

"Measuring the effectiveness of contingency arrangements is a tough challenge, not least because (like insurance policies) we hope we will never need to use them. However it makes sense to measure our investment in contingency plans and preparations, and to confirm whether management is sufficiently confident in them, prior to enacting them as by that stage it will be too late."

Possible contingency metrics suggested in the paper include:

• RTO and RPO - classic disaster recovery metrics in their own right
• Resilience - measured by incidents
• Recovery - proportions of systems for which RTP/RPO are defined, tested and met
• Costs - easier to measure than benefits, and yet an uncommon metric in practice
• Management confidence - to what extent do managers believe in the congtingency arrangements?

There are many other possible metrics in this area.  What do you measure? Why?  What do your contingency or business continuity metrics tell you?  

The art of security metrics

A security metrics opinion piece by Elden Nelson in CSO Online identifies and expands on the following four issues:

1. "Communications problems are due to a tool-centric rather than risk-centric view of security."While I accept that tool-centrism is not good, I disagree with the casual but simplistic implications that 'a tool-centric view of security' is the cause of communications problems, or that a 'risk-centric view of security' is necessarily the alternative. It sems to me there are many problems in communicating security metrics. The security or security reporting tools per se are less of an issue, in my opinion, than factors such as many technologists' fundamental misunderstandings about their own roles in the organization, about business management, strategy, risk and statistics, plus their apalling communications skills. Furthermore, communications problems are surmountable: given enough time and effort, we can get better at communicating things, putting our points across, but what is it that are we trying to communicate? That, to me, is a much more significant issue with metrics. Turning the focus of metrics from tools to risks is an improvement, but is that sufficient? I don't think it goes far enough: risks don't matter so much as risks to, and opportunities for, the organization and achievement of its business objectives. Relevance is an issue.
2. "The volume of security products in the market make seamless metrics and reporting very difficult."
   Following closely on the heels of the previous one, this issue is a red herring. Managers don't care about 'security products'. For the most part, they don't even merit a second thought, except perhaps when someone comes cap-in-hand for yet another sizeable investment in some perplexing security technology with the strong likelihood of their being nothing concrete to show for it (an inherent problem with security improvements and risk reductions). Technologists are obsessive about their tech tools, whereas managers are obsessive about the business, things such as targets and objectives, risks and opportunities, efficiencies and budgets, effectiveness and outcomes, compliance obligations, and most of all getting the most out of people, organizations and situations. The tools we use along the way are, for the most part, just the means to an end, not ends in themselves. It's not the computer screen, telephone or paper that matters but the information it conveys.
3. "Aggregate security products for seamless metrics and better communication."
   What is it with 'seamlessness'? I literally don't understand why anyone would consider the 'absense of seams' relevant to metrics, nor why aggregating products is even mentioned, while the author makes no attempt to enlighten us. The third issue falls headlong into the trap we were warned about in issue one: information security metrics aren't about security tools or products. The Mona Lisa is not a globally renowned work of art due to the astounding features of Leonardo da Vinci's palette knife.
4. "Security has moved to the central business functions—it’s no longer just an IT issue."
   
   Leaving aside the question of whether it was ever 'just an IT issue', IT security is history: today, enlightened professionals think and speak not in terms of IT security or cybersecurity but information security and information risk. The technology part is incidental, a mere commodity for the most part. Data is 'just ones and zeroes' with negligible inherent value, in direct constrast to the meaning, the knowledge, the intangible information content encoded in the numbers. The canvas beneath the Mona Lisa's image is, after all, just canvas. The paint is just paint. The physical representation of a lady sitting in a chair is largely incidental to the artwork.  Remember Magritte's "Ceci n'est pas une pipe"?

Security metrics are more representational than literal. Their purpose includes but extends well beyond the mere communication of facts. PRAGMATIC security metrics encourage their recipients to contemplate the meaning and implications for the business, leading to decisions, attitudinal shifts and (in some cases) changes to behaviours and activities. If you fail to appreciate the difference, and don't make the effort to provide relevant, topical information in a useable, meaningful form, your security metrics are doomed. Don't forget that other business information flowing around the typical corporation is, in effect, competing for the same head-space. Your security metrics need to make an impact - and, no, we're not talking about primary colors and animations, or smacking people in the head, tempting though that may be.

Management awareness paper on office information security metrics

The NoticeBored security awareness module from which we've plucked this management-level discussion paper covered information security issues relevant to the typical office or corporate workplace.

In effect, offices are information factories. Office information security controls are essential to keep the factory, its machine tools, operators and production processes running smoothly, efficiently and profitably, and to protect office-based and accessible information assets (paperwork, computer files, and white-collar workers) from all manner of risks.

Office security concerns include:

• Intruders - burglars, industrial spies and 'lost' visitors wandering loose about the place
• Fires, floods and accidents 
• Various logical/IT security incidents affecting the office network and file system, workstations, email and other applications
• Procedural issues such as workers' and visitors' failure to comply with office information security policies and procedures.

This short awareness paper outlined just a few office security metrics, without delving into details. At the time it was written (2008), we lacked the means to analyze metrics in much detail since the PRAGMATIC approach had not yet been invented. Looking back on it now, the paper is fairly typical of its day, quite naive in approach, leaving the reader to contemplate and perhaps choose between the metrics suggested.  

63,000 data points

The 2014 Data Breach Investigations Report (DBIR) by Verizon concerns more than ~63,000 incidents across 95 countries that were investigated by 50 organizations, including Verizon of course.

Fair enough ... but what exactly qualifies as an "incident"?  According to the report:

• Incident: A security event that compromises the integrity, confidentiality, or availability of an information asset. 
• Breach: An incident that results in the disclosure or potential exposure of data. 
• Data disclosure: A breach for which it was confirmed that data was actually disclosed (not just exposed) to an unauthorized party.

Those definitions are useful, although for various reason I suspect that the data are heavily biased towards IT (a.k.a. "cyber") incidents. 

~1,300 of the ~63,000 incidents were classified as breaches - an interesting metric in its own right: ~98% of incidents evidently did not result in the disclosure or potential exposure of data. For the beleaguered Chief Information Security Officer or Information Security Manager, that's a mixed blessing. On the one hand, it appears that the vast majority of incidents are being detected, processed, and presumably stopped in their tracks, without data exposure. Information security controls may have failed to prevent the 63,000 incidents, but it appears they did prevent 98% of them becoming actual breaches. That's cause for celebration, isn't it?

On the other hand, however, the 2% of incidents that actually did disclose or expose data clearly represent far more serious business impacts. Figuring out whether incidents are trivial and stoppable or are likely to become breaches is difficult at the time, hence there is little option but to respond to all incidents by default as if they are serious, resulting perhaps in a blasé attitude. 

Worse still, there is a distinct possibility that significantly more than 2% of the incidents were in fact breaches but were either not recognized or not acknowledged as such. The 2% represent abject failures of information security - hardly something that the CISO or ISM is going to admit! If they are responsible for reporting the associated metrics, these figures are dubious.  [I suspect a substantial proportion of the incidents classified as breaches were so classified because of the involvement of auditors and other independent parties, including customers and other business partners who were directly impacted and 'made a fuss'. I wonder how many purely internal breaches - breaches involving confidential business information/trade secrets as opposed to credit card numbers - were simply hushed-up and don't appear in the breach or data disclosure numbers? We won't find out from the 2014 DBIR, and to be fair we ]

Turning now to figure 16 in the report, I'm fascinated by the patterns here:

Take the second and third categories, for instance. Web App Attacks led to more than a third of breaches, yet represented only 6% of incidents. That tells me we have a serious problem with vulnerabilities in web applications. Conversely, although 18% of incidents were due to insider misuse, they caused only 8% of actual breaches - in other words, less than half of them caused real damage. The other categories in these graphs are equally interesting. Look at "cyber-espionage" for instance: only 1% of incidents caused nearly a quarter of the breaches!  [Contrary to what I said earlier, this seems to indicate that "cyber-espionage" is in fact being reported after all. Further, it points to the difficulties of being a CISO/ISM responsible for responding to and stopping such attacks, even though they are such a tiny fraction of incidents.]