Management awareness paper on network security metrics

Measuring network security involves, first and foremost, determining what 'network security' encompasses, and how it relates to the business.

Writing way back in 2007, we said that network security "comprises a range of technical and procedural controls designed to prevent, detect and/or recover from security incidents affecting the corporate data networks – incidents such as unauthorized access (hacking), worms and other malware infections, and unplanned network downtime".  The context for the paper was a NoticeBored information security awareness module exploring security arrangements protecting data networks against both deliberate and accidental threats.

The paper described ways to measure network security incidents, controls, risks, compliance and governance.  It ended with an upbeat conclusion and call-to-action: "Do not neglect the value of having the experts present and discuss reports with management.  The dialogue that ensues adds value to the written reports.  Why not present and discuss these ideas with your management and seek their opinions, bringing to the table some prototype reports in one or more formats to stimulate discussion and clarify their objectives?"

The PRAGMATIC approach is a structured, systematic way to consider the pros and cons of various metrics, leading ultimately to a decision on which ones (if any!) to adopt.  Just as important as the final destination, the method leads managers and infosec pro's together on a journey of discovery.

My guess is that the network security incident and compliance metrics would probably score above the others proposed in the paper but I can't tell for sure because I don't know a thing about your situation or measurement needs: your evaluation may well come to a markedly different conclusion.  Furthermore, in discussing the metrics paper, you might come up with 'variations on a theme', meaning variants of the metrics proposed that would score more highly, and perhaps something completely different, especially if it turns out that none of the metrics in the paper are suitable.  

That spark of creativity is the real power of PRAGMATIC.  Scientifically analyzing the factors determining the strength and suitability of the metrics under discussion leads to a better understanding of the metrics and of the measurement needs.  Contrast that with the blank-sheet approach.  Faced with the question "What network security metrics shall we adopt?", most business managers would be at a loss to know how to proceed.  If the security, network or IT people suggest one or more technical security metrics to fill the void (perhaps plucked from the air or picked out of some banale list with a pin), the managers don't have the basis for evaluating them, meaning that they are quite likely to accept whatever is offered, perhaps later coming to realize that they aren't exactly ideal.  If the metrics truly don't work, it's back to the drawing board to pick some more ... assuming someone has the good sense to call a halt to the senseless waste of time and effort (that's not a facile comment: many organizations drift aimlessly along for years with inappropriate, unsuitable or low-value metrics because everyone assumes someone else must be using them!*).

Although it is possible for the organization to develop a decent set of metrics by sheer trial-and-error, there is a distinct chance that good metrics will be discarded or discounted for arbitrary reasons, and that opportunities to 'tweak' metrics the better to fit the business needs will be missed.  Using the PRAGMATIC method as a systematic process to select, develop and maintain suitable metrics has to be a better way, don't you think?

Kind regards,

Gary

* That reminds me: have you audited your organization's security metrics? It may be a tedious assignment but a competent and diligent auditor should be able to follow the paper trail from gathering the source data through analysis and reporting to the decisions arising - if any.  If the trail goes cold, that's a strong indication that the metrics are simply not working, implying not just a waste of effort and money but an information void and lost opportunity.  Low quality metrics are a costly distraction, literally worse than useless!

Management awareness paper on malware metrics

Malware - malicious software - encompasses a variety of computer viruses, Trojans, network worms, bots and other nasties.  

Malware has been the scourge of IT users ever since the Morris worm infected the early Internet way back in 1988.  Despite the enormous global investment over the intervening years in information security controls against malware (including security awareness!), it remains a significant security concern today. 

Although antivirus software companies sometimes admit that they are fighting a losing battle, malware is generating so much income both for the VXers (malware authors) and their criminal masterminds, plus the antivirus software companies, that the arms race looks set to continue for the forseeable future.  Both sides are constantly investing in new tricks and techniques, fuelling a thriving black market in zero-day exploits and novel malware.

Meanwhile, the rest of us are lumbered with paying for it in one way or another. Addressing the malware risk is more involved than simply throwing money at antivirus software: malware metrics are the key to understanding the balance of risk against control, investing wisely, and doing our level best to keep up with, if not stay one step ahead of the game in this dynamically evolving situation.

Download our third security metrics awareness paper here.

Management awareness paper on database security metrics

The next NoticeBored security awareness paper suggests to management a whole bunch of metrics that might be used to measure the security of the organization's database systems.

Most information-packed application systems are built around databases, making database security a significant concern for the corporation.  We're talking about the crown jewels, the bet-the-farm databases containing customer, product and process information, emails, contracts, trade secrets, personal data and so much more.  Despite the importance of database security, we don't know of any organization systematically measuring it ... although we do know of many that struggle to keep on top of database security design, development, testing, patching, administration and maintenance!

So how exactly are management supposed to manage database security without database security measures? Extra sensory perception, perhaps, or gut-feel? Either way, it's hardly what one might call scientific management!

Download the paper here.  We'd be fascinated in your thoughts.  Do any of these measures catch your imagination?  What other database security metrics or measurement approaches would you suggest?  What do you use?

Management awareness paper on IPR metrics

When we get a spare moment over forthcoming months, we plan to release a series of awareness papers describing metrics for a wide variety of information security topics through the SecurityMetametrics website.

The first paper, dating back to 2007, proposes a suite of information security management metrics relating specifically to the measurement of Intellectual Property Rights (IPR). Managing and ideally optimizing IPR-related controls (namely the activities needed to reduce the chances of being prosecuted by third parties for failing to comply with their copyright, patents, trademarks etc. plus those necessary to protect the organization's own IPR from abuse by others), requires management to monitor and measure them and so get a sense of the gap between present and required levels of control, apply corrective actions where necessary and improve performance going forward.

These metrics papers were originally delivered to subscribers of the NoticeBored security awareness service, as part of the management stream.  Their primary purpose is to raise awareness of the monthly topic, but really we hope to encourage information security professionals and management to think about, discuss and perhaps adopt better security metrics.  

If you follow the sequence, you'll notice our own thinking change over the 7 years since this first paper, particularly while PRAGMATIC Security Metrics was being written.  From time to time, we introduced new styles of metric, often covering the same information security topics repeatedly but from slighly different angles (there are currently 50 infosec topics in the NoticeBored portfolio, with still more to come).

If you'd like to discuss any of these papers, please comment here on the blog or through Google Plus.

Hot crazy matrix

The universal hot crazy matrix is an amusing demonstration of the power of presenting numeric data in graphical form, extracting meaningful information from the data in order to lift the discussion off the page. We shall have to include it in our security metrics course.

Non-PC sexist humour aside, the presenter's knowledge and passion for the subject are undeniable.  Contrast that enthusiastic, lively presentation with the dull, ponderous, matter-of-fact way we normally present information security and other business metrics. 'Nuff said. For more, come on the course!

Pragmatic use for a PRAGMATIC security metric

I have just published a tool developed by Ed Hodgson, Marty Carter and me to help people estimate how long their ISO/IEC 27001 ISMS implementation projects will take.

The tool is an Excel spreadsheet (DOWNLOAD). As with the remainder of the ISO27k Toolkit, it is free to use and is covered by a Creative Commons license. I will roll it into the Toolkit when the Toolkit is next updated.

The estimated project timescale depends on how you score your organization against a set of criteria - things such as the extent to which management supports the ISMS project, and its strategic fit. The scoring process uses a percentage scale with textual descriptions at four points on the scale, similar to those Krag and I described in PRAGMATIC Security Metrics. The criteria are weighted, since some are way more important than others. The scores you enter either increase or decrease the estimated timescale from a default value, using a model coded into the spreadsheets.

Ed enhanced my original model with a more sophisticated method of calculation: Ed’s version substantially extends the timescale if you score low against any criteria, emphasizing the adverse impact of issues such as limited management support and strategic fit. I have left both versions of the model in the file so you can try them both and compare them to see which works best for you … and of course you can play with the models, the criteria and the weightings as well as the scores. I suspect that Ed’s version is more accurate than mine, but maybe both are way off-base. Perhaps we have neglected some factor that you found critical?  Perhaps the weightings or the default timescale are wrong?  If you have successfully completed ISMS implementation projects, please take a look at the criteria and the models, and maybe push your numbers through to see how accurate the estimations would have been.

Feedback comments are very welcome – improvement suggestions especially – preferably on the ISO27k Forum for the benefit of the whole community, otherwise directly to me if you’re shy.

I’m afraid we haven’t yet managed to figure out how to estimate the resourcing (man-days) needed for the implementation project, as we originally planned. A couple of approaches have been suggested (such as breaking down the requirements in ISO/IEC 27001 to identify the activities and competences/skills needed) but it will take more effort to turn the suggestions into a practical tool. If you are inspired to have a go at developing a suitable tool, please make a start and I can set up another collaborative project on Google Docs to continue the development. Further general suggestions are fine but we really need something more concrete to sink our teeth into – a draft or skeleton resourcing estimator would be good. How would you go about it?

Regards,

Gary Hinson  (Gary@isect.com)  

ASIS report on security metrics

"Persuading Senior Management with Effective, Evaluated Security Metrics" is a lengthy new research report from ASIS Foundation, a membership body for (primarily) physical security professionals.

Quoting from the report's executive summary:

"Security metrics support the value proposition of an organization’s security operation. Without compelling metrics, security professionals and their budgets continue largely on the intuition of company leadership. With metrics, the security function grounds itself on measurable results that correlate with investment, and the security professional can speak to leadership in a familiar business language."

Fair enough.  That's similar to what we wrote in PRAGMATIC Security Metrics, in referring to measurable results and business orientation.

"Security metrics are vital, but in the field and in the literature one finds few tested metrics and little guidance on using metrics effectively to inform and persuade senior management." 

I'm not sure I agree that there are 'few tested security metrics' - it all depends on what we mean by 'tested' and 'security metrics'. I know there are hundreds of information security metrics in use, scores of which I believe are relatively widespread. I also know how easy it is to specify literally thousands of potential information security metrics covering various aspects and facets of our field, especially if one considers variants of any given metric as distinct metrics (e.g. 'the malware threat' could be measured in dozens of different ways, and each of those measures could be expressed or reported in dozens of ways, implying several gross of malware threat metrics).

We described and evaluated about 150 information security metrics examples in PRAGMATIC Security Metrics, and mentioned or hinted at numerous variants that might address some of the shortcomings we found in the examples.

"To address the gap, in spring 2013 the ASIS Foundation sponsored a major research project designed to add to the body of knowledge about security metrics and to empower security professionals to better assess and present metrics. The Foundation awarded a grant to Global Skills X-change (GSX), partnered with Ohlhausen Research, to carry out the project." 

GSX, tagline "Define. Measure. Optimize.", describes itself as "... a professional services firm that specializes in designing workforce education strategies and processes, which allow customers to meet their specific performance goals. The GSX core business model revolves around defining functional competency models and developing valid and reliable assessment tools as the foundation of credentialing and educational programs."

As to Olhausen Research, "A researcher in the security field for more than 25 years, [Peter Ohlhausen, President of Olhausen Research Inc.] has assisted in the multi-year revision of Protection of Assets, served as senior editor of Security Management magazine, and conducted numerous research and consulting projects for the U.S. Department of Justice, U.S. Department of Homeland Security, ASIS, and corporate clients." 

"This report provides the project’s findings, including its three practical, actionable products:

• The Security Metrics Evaluation Tool (Security MET), which security professionals can self-administer to develop, evaluate, and improve security metrics 

• A library of metric descriptions, each evaluated according to the Security MET criteria 

• Guidelines for effective use of security metrics to inform and persuade senior management, with an emphasis on organizational risk and return on investment"

Security MET turns out to be a method for assessing and scoring metrics according to 9 criteria in 3 categories, described in some detail in Appendix A of the ASIS report:

Technical Criteria – Category 1

1. Reliability

2. Validity

3. Generalizability

Operational (Security) Criteria – Category 2

4. Cost 

5. Timeliness

6. Manipulation

Strategic (Corporate) Criteria – Category 3

7. Return on Investment

8. Organizational Relevance

9. Communication

I see interesting parallels to the 9 PRAGMATIC criteria we have described. Security MET requires users to score metrics on a 1-5 scale against each criterion using a 3 point scoring scale rather than the percentage scales with 4 scoring points that we prefer, but is otherwise the same process. It appears that we have converged on a generalized method for assessing or evaluating metrics. However, PRAGMATIC Security Metrics, and indeed other metrics books, are notably lacking from their "comprehensive review of the current state of metric development and application".  Information security is barely even mentioned in the entire report, an unfortunate omission given the convergence of our fields.

The ASIS report goes on to list just 16 [physical] security metrics, described as "authentic examples" that had been identified through the researchers' telephone interviews with respondents:

1. Office Space Usage Metric

2. Security Activity Metric

3. Environmental Risk Metric

4. Averted External Loss Metric

5. Security Audit Metric

6. Officer Performance Metric Panel

7. Security-Safety Metric

8. Security Incidents Metric

9. Personnel Security Clearance Processing Metric

10. Loss Reduction/Security Cost Metric

11. Operations Downtime Reduction Metric

12. Due Diligence Metric

13. Shortage/Shrinkage Metric

14. Phone Theft Metric

15. Security Inspection Findings Metric

16. Infringing Website Compliance Metric

These metrics have each been MET-scored by a handful of people, potentially generating statistics concerning consistency or reliability of the scoring method although the statistics are not actually provided in the report. There is no information reported concerning the method's consistency if the assessments are repeated by the same assessors. The authors do however mention that "The total score may suggest how close the metric is to attaining the highest possible score (45), but it is not likely to be useful for comparing different metrics, as the scoring would be different for users in different organizations.", suggesting that they are unhappy with the method's consistency between different assessors.  

Overall, this report is a worthwhile contribution to the security metrics literature.  Take a look!