Pragmatic use for a PRAGMATIC security metric

I have just published a tool developed by Ed Hodgson, Marty Carter and me to help people estimate how long their ISO/IEC 27001 ISMS implementation projects will take.

The tool is an Excel spreadsheet (DOWNLOAD). As with the remainder of the ISO27k Toolkit, it is free to use and is covered by a Creative Commons license. I will roll it into the Toolkit when the Toolkit is next updated.

The estimated project timescale depends on how you score your organization against a set of criteria - things such as the extent to which management supports the ISMS project, and its strategic fit. The scoring process uses a percentage scale with textual descriptions at four points on the scale, similar to those Krag and I described in PRAGMATIC Security Metrics. The criteria are weighted, since some are way more important than others. The scores you enter either increase or decrease the estimated timescale from a default value, using a model coded into the spreadsheets.

Ed enhanced my original model with a more sophisticated method of calculation: Ed’s version substantially extends the timescale if you score low against any criteria, emphasizing the adverse impact of issues such as limited management support and strategic fit. I have left both versions of the model in the file so you can try them both and compare them to see which works best for you … and of course you can play with the models, the criteria and the weightings as well as the scores. I suspect that Ed’s version is more accurate than mine, but maybe both are way off-base. Perhaps we have neglected some factor that you found critical?  Perhaps the weightings or the default timescale are wrong?  If you have successfully completed ISMS implementation projects, please take a look at the criteria and the models, and maybe push your numbers through to see how accurate the estimations would have been.

Feedback comments are very welcome – improvement suggestions especially – preferably on the ISO27k Forum for the benefit of the whole community, otherwise directly to me if you’re shy.

I’m afraid we haven’t yet managed to figure out how to estimate the resourcing (man-days) needed for the implementation project, as we originally planned. A couple of approaches have been suggested (such as breaking down the requirements in ISO/IEC 27001 to identify the activities and competences/skills needed) but it will take more effort to turn the suggestions into a practical tool. If you are inspired to have a go at developing a suitable tool, please make a start and I can set up another collaborative project on Google Docs to continue the development. Further general suggestions are fine but we really need something more concrete to sink our teeth into – a draft or skeleton resourcing estimator would be good. How would you go about it?

Regards,

Gary Hinson  (Gary@isect.com)  

ASIS report on security metrics

"Persuading Senior Management with Effective, Evaluated Security Metrics" is a lengthy new research report from ASIS Foundation, a membership body for (primarily) physical security professionals.

Quoting from the report's executive summary:

"Security metrics support the value proposition of an organization’s security operation. Without compelling metrics, security professionals and their budgets continue largely on the intuition of company leadership. With metrics, the security function grounds itself on measurable results that correlate with investment, and the security professional can speak to leadership in a familiar business language."

Fair enough.  That's similar to what we wrote in PRAGMATIC Security Metrics, in referring to measurable results and business orientation.

"Security metrics are vital, but in the field and in the literature one finds few tested metrics and little guidance on using metrics effectively to inform and persuade senior management." 

I'm not sure I agree that there are 'few tested security metrics' - it all depends on what we mean by 'tested' and 'security metrics'. I know there are hundreds of information security metrics in use, scores of which I believe are relatively widespread. I also know how easy it is to specify literally thousands of potential information security metrics covering various aspects and facets of our field, especially if one considers variants of any given metric as distinct metrics (e.g. 'the malware threat' could be measured in dozens of different ways, and each of those measures could be expressed or reported in dozens of ways, implying several gross of malware threat metrics).

We described and evaluated about 150 information security metrics examples in PRAGMATIC Security Metrics, and mentioned or hinted at numerous variants that might address some of the shortcomings we found in the examples.

"To address the gap, in spring 2013 the ASIS Foundation sponsored a major research project designed to add to the body of knowledge about security metrics and to empower security professionals to better assess and present metrics. The Foundation awarded a grant to Global Skills X-change (GSX), partnered with Ohlhausen Research, to carry out the project." 

GSX, tagline "Define. Measure. Optimize.", describes itself as "... a professional services firm that specializes in designing workforce education strategies and processes, which allow customers to meet their specific performance goals. The GSX core business model revolves around defining functional competency models and developing valid and reliable assessment tools as the foundation of credentialing and educational programs."

As to Olhausen Research, "A researcher in the security field for more than 25 years, [Peter Ohlhausen, President of Olhausen Research Inc.] has assisted in the multi-year revision of Protection of Assets, served as senior editor of Security Management magazine, and conducted numerous research and consulting projects for the U.S. Department of Justice, U.S. Department of Homeland Security, ASIS, and corporate clients." 

"This report provides the project’s findings, including its three practical, actionable products:

• The Security Metrics Evaluation Tool (Security MET), which security professionals can self-administer to develop, evaluate, and improve security metrics 

• A library of metric descriptions, each evaluated according to the Security MET criteria 

• Guidelines for effective use of security metrics to inform and persuade senior management, with an emphasis on organizational risk and return on investment"

Security MET turns out to be a method for assessing and scoring metrics according to 9 criteria in 3 categories, described in some detail in Appendix A of the ASIS report:

Technical Criteria – Category 1

1. Reliability

2. Validity

3. Generalizability

Operational (Security) Criteria – Category 2

4. Cost 

5. Timeliness

6. Manipulation

Strategic (Corporate) Criteria – Category 3

7. Return on Investment

8. Organizational Relevance

9. Communication

I see interesting parallels to the 9 PRAGMATIC criteria we have described. Security MET requires users to score metrics on a 1-5 scale against each criterion using a 3 point scoring scale rather than the percentage scales with 4 scoring points that we prefer, but is otherwise the same process. It appears that we have converged on a generalized method for assessing or evaluating metrics. However, PRAGMATIC Security Metrics, and indeed other metrics books, are notably lacking from their "comprehensive review of the current state of metric development and application".  Information security is barely even mentioned in the entire report, an unfortunate omission given the convergence of our fields.

The ASIS report goes on to list just 16 [physical] security metrics, described as "authentic examples" that had been identified through the researchers' telephone interviews with respondents:

1. Office Space Usage Metric

2. Security Activity Metric

3. Environmental Risk Metric

4. Averted External Loss Metric

5. Security Audit Metric

6. Officer Performance Metric Panel

7. Security-Safety Metric

8. Security Incidents Metric

9. Personnel Security Clearance Processing Metric

10. Loss Reduction/Security Cost Metric

11. Operations Downtime Reduction Metric

12. Due Diligence Metric

13. Shortage/Shrinkage Metric

14. Phone Theft Metric

15. Security Inspection Findings Metric

16. Infringing Website Compliance Metric

These metrics have each been MET-scored by a handful of people, potentially generating statistics concerning consistency or reliability of the scoring method although the statistics are not actually provided in the report. There is no information reported concerning the method's consistency if the assessments are repeated by the same assessors. The authors do however mention that "The total score may suggest how close the metric is to attaining the highest possible score (45), but it is not likely to be useful for comparing different metrics, as the scoring would be different for users in different organizations.", suggesting that they are unhappy with the method's consistency between different assessors.  

Overall, this report is a worthwhile contribution to the security metrics literature.  Take a look!

Another day, another survey, another ten failures

An article in an eZine concerning a security survey by PwC, sponsored by Iron Mountain, caught my eye today because they offer to benchmark respondents against others. So, purely in the interest of metrics research, I had a go at the benchmark tool.

First of all, the tool asked me for an email address without explaining why. Fail #1 (see also #6 below).

Thankfully, the email address validation routine is easily fooled. Fail #2 (or possibly Success #1 depending on one's perspective!).

Next the survey asked about 20 questions, mostly lame and some badly worded. There is no explanation about why those 20 questions have been selected. They address only a small part of information security. Fail #3.

All 20 questions have the same set of 4 possible multiple-choice answers, even though the stock answers don't cover all possibilities and don't even make sense for all the questions. The survey design is poor. Fail #4.

At the end of the survey, I was presented with a comparative "index score", in my case 41 (presumably 41%), along with a nasty day-glow bar chart and the following commentary:

"Your risk level is serious. Your score is well below the PwC recommended threshold, and it is only a matter of time before serious problems occur. Your business needs to take action now to improve information security and working practices. Read this report for further insight into your business risk and gain practical advice on how you can increase your index score and reduce your risk."

Our risk level is 'serious', eh?  Spot the scare tactics! FUD! Thanks to those 20 lame questions, they know next to nothing about our true situation, yet they presume to tell me we are "well below the PwC recommended threshold". Bloody cheek! Yes, we are facing various information security threats and have various information security vulnerabilities, and no, we have not implemented all the information security controls that might be appropriate for other organizations, but that's not the same as saying we are at serious risk. As my pal Anton would say, "Context is everything". Fail #5.

Next I was asked for yet more personal information in order to access the 'personalized report'. In reality, of course, this is clearly a marketing initiative so I know what they are up to, though again they don't actually say. Failing to explain why the information is needed conflicts with at least one of the OECD privacy principles concerning personal data collection and I think would be illegal under the privacy laws in most of the world outside the US. Fail #6.  

And again the data entry validation routines are weak. Fail #7.

The 'personalized report', "Your information risk profile", compares my score against "averages" (mean scores?) from the PwC/Iron Mountain survey in a PDF report. Generating the PDF on the fly is cool ... but the actual content is poor. The scope and purpose of the PwC survey are not stated in the 'personalized report', nor is the sample size or other basic information about the survey methods. The entire basis of the benchmarking is dubious, particularly if the PwC survey that generated the comparative data also used the lame 20 question multi-choice method. It's essentially meaningless drivel. Fail #8.

For no obvious reason, the 'personalized report' includes a page stating 4 "worrying facts", the first of which being "88% consider paper to be the biggest threat to information security". Eh? In all my years of information security risk management, I have NEVER heard paper being described as an information security threat. Paper is an asset usually of negligible value, although the information content on paperwork can be extremely valuable and a few, very rare bits of paper are priceless ... oh, hang on, Iron Mountain sponsored this survey, right. Ah yes, I remember, that's the same Iron Mountain whose archive facilities have suffered several serious fires including one recently in Buenos Aires. So much for their security credentials. It could be argued that Iron Mountain is "the biggest threat to [its customers'] information security"! Fail #9.

The 'key findings - your next steps' to the 'personalized report' kind of make sense in so far as they go, but bear no obvious relation to the benchmark, survey or the data.  Although for example I'm personally in favor of step 1 'Take it to the top - Get board level support by taking a strategic approach to information management', I'm also in favor of 'Don't run with scissors' and 'Don't smoke' which make about as much sense in this context. Fail #10.

As to the actual PwC/Iron Mountain survey, I encourage you to take a critical look at the survey report, and make of it what you will. Read past the annoying repetitive references to "the mid market" (which I think must be marketing-speak for medium-sized organizations) and the wrongly-labeled graphs. Set aside the spurious references to additional information and news headlines muddled in with the survey data, and the buzzwords-du-jour. Consider the "comprehensive questionnaire" of just 34 statements and the dubious statistics arising, and see what you have left.

Then read the report's disclaimer very carefully:

"This publication has been prepared for general guidance on matters of interest only, and does not constitute professional advice. You should not act upon the information contained in this publication without obtaining specific professional advice. No representation or warranty (express or implied) is given as to the accuracy or completeness of the information contained in this publication, and, to the extent permitted by law, PricewaterhouseCoopers LLP, its members, employees and agents do not accept or assume any liability, responsibility or duty of care for any consequences of you or anyone else acting, or refraining to act, in reliance on the information contained in this publication or for any decision based on it."

[PwC, I'm very disappointed in you. Your work on the UK DTI/BERR security surveys has been pretty good. Your consultants and auditors are generally competent and well-respected. What on Earth possessed you to get involved in this nonsense? Are you really that hard up these days?]

Security metrics books

Dell security analyst Ben Knowles has reviewed and compared four information security metrics books:

• Andrew Jaquith's Security Metrics (aka "the Treefrog book"!)
• Caroline Wong's Security Metrics
• Lance Hayden's IT Security Metrics
• and ours, PRAGMATIC Security Metrics

Ben's comments are sound: while these books present differing perspectives and messages, all four have merit.  We discussed the first three books (and more) in the literature review in PRAGMATIC Security Metrics, and on SecurityMetametrics.com

Enterprise Security Metrics report

A new 28-page research report by George Campbell's Security Executive Council (SEC) concerns the status of physical security metrics. Enterprise Security Metrics: A Snapshot Assessment of Practices (free but registration required) "provides a snapshot of the use of metrics in corporate security management. It includes information on the current state-of-the-art of various models of benchmarking and security metrics, types of metrics, judging the maturity of security metrics programs as well as challenges and opportunities for those undertaking security metrics programs. This report specifically summarizes our learned experience from corporate security measures and metrics initiatives."

The report refers to SEC's ongoing metrics research but unfortunately does not go into details about the methods.  A note on page 7 refers to a survey of 27 companies representing "a solid cross section of industry sectors [with] mature and multi-service corporate security programs, several engaging in best practice operations". The small sample was presumably drawn from members or clients of the SEC meaning that it was not random but self-selected from organizations with a clear interest in security metrics. Nevertheless, statistics aside, the findings and conclusions are well worth reading in more general terms - for example:

"Nearly 70 percent of respondents stated that they don’t collect security program metrics for the purposes of presenting to senior management ... This lack of engagement remains as a significant internal obstacle to metrics acceptance and development. Too many corporate security practitioners have either avoided or failed to understand the relevance of such measures. Security organizations have the data; they are willing to count events and other activity data but they apparently don’t see the need to use it to build actionable, influential metrics that can effectively influence senior management."

I like the phrase 'actionable, influential metrics'. Metrics that are neither actionable nor influential have little practical value. They are "coffee table metrics", the sorts of things one might idly skim through in a glossy magazine.  Metrics that are influential but not actionable can cause consternation: we know there is something wrong but we don't know how to fix it. Metrics that are actionable but not influential have no impact. Metrics that don't influence or support decisions are essentially pointless. Fow what it's worth, most such metrics tend to be cheap and easy to gather so the measurement costs are quite low, although there is a hidden cost in that they can be distracting, giving the impression that someone is on top of security metrics whereas in fact they are not.

The report mentions the commonplace KRIs and KPIs (Key Risk and Performance Indicators) plus two metrics that were new to me: Key Influence Indicators ("How do our metrics influence governance policy, business unit accountability and personal behavior?") and Key Value Indicators ("How have our metrics demonstrated tangible, actionable and measurable benefit to the enterprise?"). Influence and value are two of several characteristics of metrics, or metametrics. The PRAGMATIC method uses nine specific metametrics to determine the net value of a metric.

There is a common theme underlying the report's conclusions, namely that more effort should be put into identifying baseline metrics for all aspects of security in order to enable benchmarking comparisons between organizations. Security management practices and metrics requirements vary widely largely in practice because security risks vary widely, hence the particular security concerns that drive a given organization to select specific security metrics may not coincide with other organizations. However, an appendix to the SEC report offers a maturity metric measuring the status of an organization's [physical] security metrics program by assessing the anticipated parts of such a program. The metric is similar in style to those we described in PRAGMATIC Security Metrics, a form of metric that encourages us to break down and systematically assess complex situations within the organization (I found them well suited for internal audits and process improvement initiatives). Maturity metrics are also a promising approach for benchmarking comparisons of multiple organizations.

Another conclusion of the report is that metrics are needed for compliance assessment purposes: we discussed this point too in PRAGMATIC Security Metrics. Industry regulators and authorities (such as the other SEC!) need rational ways in which to measure and assess organizations on all sorts of criteria including governance, risk and security practices. The conventional approach is to specify and mandate certain requirements, in which case the measurement process boils down to someone (hopefully, a competent, independent and diligent third party) determining whether the stated requirements have or have not been fulfilled - fine in theory but harder to achieve in practice since there are so many variables. PCI-DSS, for instance, requires a number of specific security controls supposedly to secure cardholder data, and PCI assessments attempt to confirm that they are all in place. We know from Target and many other breaches that the PCI controls are imperfect, and that a "pass" on the PCI assessment does not necessarily mean that card holder data are in fact adequately secured. Furthermore, card holder data are just a fraction of most organizations' information assets, hence ticking the PCI compliance box does not necessarily mean the organization has adequate information security as a whole (it is an indicator, perhaps, but primarily concerns compliance with externally-imposed obligations). It would be practically impossible to extend the PCI-type approach to cover all of information security, physical security, risk management and governance, whereas other approaches such as maturity metrics could be used both to measure and to drive improvements.

George ends the report with a plea to collaborate with other metrics professionals. I welcome the initiative and will definitely get in touch! 

What are KPIs?

Krag and I have discussed this question from time to time and, although we are broadly aligned in our thinking, we haven't yet totally resolved our differences ... which makes the exchanges fun.

With that in mind, I always wonder what someone really means when they talk about KPIs. To some, Key Performance Indicator has a very specific and particular meaning, although I suspect if we assembled a dozen such people in a room to discuss it, we'd soon end up realizing that we have more than a dozen different interpretations! 

To others (including me, as it happens), KPI is a generic, blanket term for a class or type of metric that satisfies the criteria implied by the term: 

• Key implies that the metric itself is especially important, crucial or vital even, given that there are many many different ways to measure and assess things but most of them are of limited value. Picking out the few things that truly matter is a core issue in metrics. 'Spam volume' is an example of a metric that is both narrow and shallow, whereas 'email risk level' is a much broader, deeper and richer metric, and is far more likely to be considered key (even if we happen to be talking specifically about metrics relating to spam filtering). However, the criticality and value of metrics does depend on the contexts or situations being measured and the perspectives and information needs of their users. It is conceivable that 'spam volume' may be considered a KPI for the anti-spam controls, but that's a narrow perspective. Key may also refer to the performance, in the sense that the KPI is an indicator concerning an important issue; 
• Performance is a distinctly ambiguous word, implying concern for the process and/or its outcome. Are we measuring key activities (typically in order to assess and improve the efficiency of the process) or its outcomes (typically to assess and improve the effectiveness of the process), neither or both? I have seen KPI used in several different senses, although usually it is not totally clear (perhaps not even to the person discussing it!) which one is meant; 
• Indicator generally means simply a metric or measure but it may imply imprecision or approximation. A typical car's fuel gage, for instance, gives a fairly vague indication of the amount of fuel in the tank, whereas the equivalent metric on an aircraft tends to be much more precise, accurate and reliable, for obvious reasons. The car's fuel gage may not tell you how many litres remain but if it heads into the red zone, you know you need to find a filling station soon. Indicator often also implies a forward-looking or predictive rather than purely historical measure. 'Trends' are common examples, used to manage various aspects of information security where precision is nice but not vital (e.g. supporting security investment or resourcing decisions). 

As far as I'm concerned, then, a KPI is generally a predictive metric concerning some critical outcome of an important process. In the information security context, KPIs are most likely to measure core security processes such as risk assessment, and key controls such as authentication and access control. Efficiency is secondary to effectiveness, in that security failures resulting from ineffective controls can lead to serious and potentially very damaging incidents, whereas inefficient controls are merely a bit wasteful (that's actually a strong bias, one worth challenging in situations where security becomes onerous for information users and administrators, perhaps so onerous that they bypass or disable the controls sending us back to square one!).

Avoiding metrics myopia

Things being measured are patently being observed in a very specific, focused manner. Things that are observed so closely tend to be presumed by others to be of concern to the measurer/observer, at least. Things under the measurement ruler therefore assume a certain significance simply by dint of their being closely observed, regardless of any other factors.

We see this 'fascination bias' being actively exploited by cynical promoters, marketers and advertisers on a daily basis through an endless stream of largely banal and unscientific online polls and infographics, their true purpose made all the more obvious by the use of bright primary-color eye-catching graphics. They are manipulating readers to believe that since something has been measured, it must be important. How many of us take the trouble to think about the quality of the metrics, or about all the other aspects that haven't been measured? Like bunnies in the headlights, we stare at the numbers.

In PRAGMATIC Security Metrics, we outlined 21 observer biases drawn from an even longer list drawn up by Kahneman, Slovic and Tversky in Judgment Under Uncertainty (1982): what I'm calling 'fascination bias' has some resemblance to what Kahneman et al. described as 'attentional bias', the tendency to neglect relevant data when making judgments of a correlation or association.

Fascination bias creates a genuine concern in that we tend to measure things that are relatively easy to measure, and place undue faith in those metrics relative to other factors that are not being measured. Back in 2011, Michel Zalewski said in his blog:

"Using metrics as long-term performance indicators is a very dangerous path: they do not really tell you how secure you are, because we have absolutely no clue how to compute that. Instead, by focusing on hundreds of trivial and often irrelevant data points, they take your eyes off the new and the unknown."

While we don't entirely accept that we 'have no clue how to compute security performance', his point about neglecting other risks and challenges due to being inordinately focused on specific metrics is sound.  It's only natural that what gets measured gets addressed (though not necessarily improved!). The unfortunate corollary is that what doesn't get measured gets neglected.

The upshot of this is that there is a subtle obligation on those who choose metrics to find ways to measure all the important matters, even if some of those metrics are expensive/complex/qualitative/whatever. It's simply not good enough to measure the easy stuff, such as the numbers that assorted security systems constantly pump out 'for free'. It's inappropriate to disregard harder-to-measure issues such as culture, ethics, awareness and trust, just as it is inappropriate to restrict your metrics to IT or cybersecurity rather than information security.

That's one of the key reasons why we favor the systematic top-down GQM approach: if you start by figuring out the Goals or objectives of security, expand on the obvious Questions that arise and only then pick out a bunch of potential Metrics to answer those questions, it's much harder to overlook important factors. As to figuring out the goals or objectives, conceptual frameworks for information security such as BMIS and ISO27k, based on fundamental principles, are an obvious way to kick-off the thinking process and frame the initial discussion.