Metrics thought for the day

Where relevant, using current business metrics (also) for information risk and security purposes can be cost-effective if suitable raw data are already being gathered: the additional analysis, reporting and use incur relatively little incremental cost, especially if largely automated.

Corollary: when searching for metrics in any area of information risk and security, don't forget to check through existing business metrics alread in use for anything suitable, either as-is or with minor changes.

It would be easier to identify such metrics if the organization maintained a metrics inventory or database ...

Decision-led metrics

Metrics in general are valuable because, in various ways, they support decisions. If they don't, they are at best just nice to know - 'coffee table metrics' I call them. If coffee table metrics didn't exist, we probably wouldn't miss them, and we'd have cut costs.

So, what decisions are being, or should be, or will need to be made, concerning information risk and security? If we figure that out, we'll have a pretty good clue about which metrics we do or don't want.

Here are a few ways to categorize decisions:

• Decisions concerning strategic, tactical and operational matters, with the corresponding long, medium and short-term focus and relatively broad, middling or narrow scope;
• Decisions about risk, governance, security, compliance ...;
• Decisions about what to do, how to do it, who does it, when it is done ...;
• Business decisions, technology decisions, people decisions, financial decisions ...;
• Decisions about departments, functions, teams, systems, projects, organizations; 
• Decisions regarding strategies/approaches, policies, procedures, plans, frameworks, standards ...;
• Decisions relating to threats, vulnerabilities and impacts - evaluating and responding to them;
• Decisions made by senior, middle or junior managers, by staff, and perhaps by or relating to business partners, contractors and consultants, advisors, stakeholders, regulators, authorities, owners and other third parties;
• Decisions about effectiveness, efficiency, suitability, maturity and, yes, decisions about metrics (!);
• ... [feel free to bring up others in the comments].

Notice that the bullets are non-exclusive: a single metric might support strategic decisions around information risks in technology involving a commercial cloud service, for instance, putting it in several of those categories. 

If we systematically map out our current portfolio of security metrics (assuming we can actually identify them: do we even have an inventory or catalog of security metrics?) across all those categories, we'll probably notice two things. 

First, for all sorts of reasons, we will probably find an apparent excess or surplus of metrics in some areas and a dearth or shortage elsewhere. That hints at perhaps identifying and developing additional metrics in some areas, and cutting down on duplicates or failing/coffee-table metrics where there seems to be too many which is itself a judgement call or a decision about metrics - and not as obvious as it may appear. Simplistically aiming for a "balance" of metrics across the categories is a naive approach

Second, some metrics will pop up in multiple categories ... which is wonderful. We've just identified key metrics. They are more important than most since they evidently support multiple decisions. We clearly need to be extra careful with these metrics since data, analysis or reporting issues (such as errors and omissions, or unavailability, or deliberate manipulation) is likely to affect multiple decisions.

Overall, letting decisions and the associated demand for information determine the organization's choice of metrics makes a lot more sense than the opposite "measure everything in sight" data-supply-driven approach. What's the point in measuring stuff that nobody cares about? 

Metrics database

I wonder if any far-sighted organizations are using a database/systems approach to their metrics? Seems to me a logical approach given that there are lots of measurement data swilling around the average corporation (including but not only those relating to information risk, security, control, governance, compliance and privacy). Why not systematically import the data into a metrics database system for automated analysis and presentation purposes? Capture the data once, manage it responsibly, use it repeatedly, and milk the maximum value from it, right?

If you think that's a naive, impracticable or otherwise krazy approach, please put me straight. What am I missing? Why is it that I never seem to hear about metrics databases, other than generic metrics catalogs (which are of limited value IMNSHO) and Management Information Systems (which were all the rage in the 80s but strangely disappeared from sight in the 90s)?

Conversely, if your organization has a metrics database system, how is it working out in practice?  What can you share with us about the pros and cons?

Security dashboard tips

Tripwire blog's The Top 10 Tips for Building an Effective Security Dashboard is an interesting collection of advice from several people. It's thought provoking, although I don't entirely agree with it.

Tip 2 'Sell success, not fear', mentions:

"For example, in the event that they cannot find personnel who come equipped with the skills needed to improve progress, security personnel can use dashboards to demonstrate the impact that well trained individuals could have on finding and resolving issues and threats, as well as to subsequently leverage that insight for training and cultivating available skills."

Although somewhat manipulative, metrics can indeed provide data supporting or justifying proposed security improvements, assuming that, somehow, someone has already decided what needs to be done ... and suitable metrics can be useful for that purpose too.

The thrust of tip 4 'Use compelling visualizations' is that the dashboard needs to be glossy: I agree dashboards should be professionally crafted and reasonably well presented but I feel their true value and utility has far more to do with the information content than the look.

Tip 9 'Thoroughly vet the information before it is presented' is an odd one. The advice to be ready to explain outliers and anomalies makes sense, but the implication of someone vetting the data before it goes to the dashboard is that it will be both delayed and sanitized. Hmmm.

Well, take a look for yourself and see what you make of the ten tips.

Metrics case study on Boeing

The Security Executive Council has published an interesting case study concerning the review and selection of metrics relating to physical and information risks at Boeing.  [Access to the article is free but requires us to register our interest.]

The case study mentions using SMART criteria and a few other factors to select metrics but doesn't go into details, unfortunately.  Nevertheless, the analytical approach is worth reading and contemplating.

If we were to conduct such an assignment for a client today, we would utilize a combination of tools and techniques across six distinct phases:

1. Background information gathering concerning Boeing's business situation, information risks, and existing metrics, using standard analytical or audit methods, clarifying the as-is situation and building a picture of what needs to change, and why. This phase would typically culminate in a report and a presentation/discussion with management.
   
   
2. GQM (Goal-Question-Metric) assessment eloquently described by Lance Hayden in IT Security Metrics. This is a more structured and systematic version of the approach outlined in the case study. A workshop approach would be useful, probably several in fact to delve into various aspects with the relevant business people and experts. The output would be a matrix or tree-root diagram illustrating the goals, questions and metrics.
   
   
3. PRAGMATIC assessment and ranking of the metrics generated in phase 2 using the approach documented in our book. The output would be a management report containing a prioritized list of metrics ranked according to their PRAGMATIC scores, leading to a further presentation/discussion with management and, hopefully, agreement on a shortlist of the most promising metrics, those actually worth pursuing. This and the previous phase would take a creative approach, thinking about what needs to be measured, why, how, when etc., using both GQM and PRAGMATIC to firm-up the metrics that best fit the requirements  and focus groups to finalize the metrics (both existing metrics that are worth retaining possibly with some changes, and novel metrics being introduced).
   
4. Planning and preparing for the implementation phase, perhaps including pilot studies.
   
   
5. Implementation: making the changes needed to collect, analyse, report and most of all use the metrics.  This might well involve retiring or recasting some of the client's existing metrics that haven't earned their keep, in a way that teases out the last dregs of value from the data gathered previously.
   
6. Ongoing metrics management and maintenance: using information from the GQM and PRAGMATIC steps to monitor and if appropriate refine or replace the metrics, ensuring for instance that they are proving valuable to the business (i.e. they should be cost-effective - one of the PRAGMATIC criteria conspicuously absent from SMART).  

In parallel with that sequence would be conventional project management activities - planning, resourcing & team building, motivation, tracking, reporting and assignment risk management.

Get in touch to review and update your metrics: we'd love to help!

Lean security

Lean manufacturing or kaizen is a philosophy or framework comprising a variety of approaches designed to make manufacturing and production systems as efficient and effective as possible, approaches such as:

• Design-for-life - taking account of the practical realities of production, usage and maintenance when products are designed, rather than locking-in later nightmares through the thoughtless inclusion of elements or features that prove unmanageable;
• Just-in-time delivery of parts to the production line at the quantity, quality, time and place they are needed (kanban), instead of being stockpiled in a warehouse or parts store, collecting dust, depreciating, adding inertia and costs if product changes are needed;
• Elimination of waste (muda) - processes are changed to avoid the production of waste, or at the very least waste materials become useful/valuable products, while wasted time and effort is eliminated by making production processes slick with smooth, continuous, even flows at a sensible pace rather than jerky stop-starts;
• An obsessive, all-encompassing and continuous focus on quality assurance, to the extent that if someone spots an issue anywhere on the production line, the entire line may be stopped in order to fix the root cause rather than simply pressing ahead in the hope that the quality test and repair function (a.k.a. Final Inspection or Quality Control) will bodge things into shape later ... hopefully without the customer noticing latent defects;
• Most of all, innovation - actively seeking creative ways to bypass/avoid roadblocks, make things better for all concerned, and deliver products that go above and beyond customer expectations, all without blowing the budget.

Service industries and processes/activities more generally can benefit from similar lean approaches ... so how might kaizen be applied to information risk management and security?

• Design-for-security - products and processes should be designed from the outset to take due account of information security and privacy requirements throughout their life, implying that those requirements need to be elaborated-on, clarified/specified and understood by the designers;
• Just-in-case - given that preventive security controls cannot be entirely relied-upon, detective and corrective controls are also necessary;
• Elimination of doubt - identifying, characterizing and understanding the risks to information (even as they evolve and mutate) is key to ensuring that our risk treatments are necessary, appropriate and sufficient, hence high-quality, reliable, up-to-date information about information risk (including, of course, risk and security metrics) is itself an extremely valuable asset, worth investing in;
• Quality assurance applies directly - information security serves the business needs of the organization, and should be driven by risks of concern to various stakeholders, not just 'because we say so';
• Innovation also applies directly, as stated above.  It takes creative effort to secure things cost-effectively, without unduly restricting or constraining activities to the extent that value is destroyed rather than secured.

Smoke-n-mirrors IBM style

I've just been reading the IBM 2015 Cyber Security Intelligence Index, trying to figure out their 'materials and methods' i.e. basic parameters for the survey, such as population size and nature. All I can find are some obtuse references in the first paragraph:

"IBM Managed Security Services continuously monitors billions of events per year, as reported by more than 8,000 client devices in over 100 countries. This report is based on data IBM collected between 1 January 2014 and 31 December 2014 in the course of monitoring client security devices as well as data derived from responding to and performing analysis on cyber attack incidents. Because our client profiles can differ significantly across industries and company size, we have normalized the data for this report to describe an average client organization as having between 1,000 and 5,000 employees, with approximately 500 security devices deployed within its network."

Reading between the lines, it appears that this is a report gleaned primarily from 'more than 8,000 client [network security?] devices' belonging to an unknown number of organizations around the world who are customers of IBM Managed Security Services ... which IBM has described as:

"24/7/365 monitoring and management of security technologies you house in your environment. IBM provides a single management console and view of your entire security infrastructure, allowing you to mix and match by device type, vendor and service level to meet your individual business needs while drastically reducing your security costs, simplifying security management and accelerating your speed to protection."

But, before you delve into the actual report, read that final sentence of the first paragraph again: they have 'normalized the data' (whatever that means) to an 'average client organization' with about 500 security devices ... so given the total of 8,000 devices, and on the assumption that 'average' means 'mean', it appears the survey covers just 16 organizations whose network security devices are managed by IBM. Oh boy oh boy. No wonder they are so reluctant to tell us about the analytical methods!  

The data are from 2014, the report was published in July 2015. Given the miniscule sample, I wonder why it took them 7 months to do the analysis and reporting? Crafting the words to gloss over the glaring flaws, perhaps?

The remainder of the report is pretty humdrum - some superficially interesting graphics and four 'case studies' (three of which - that's 75% or a 'vast majority', IBM - are not actual cases as such but fictional accounts based on the collective experiences of an unknown number of clients). There's nothing particularly unusual or noteworthy in the report, despite the hyperbole (2014 was hardly "The year the Internet fell apart", IBM). The trends and other statistical information is worthless in scientific terms.

Remember this cynical blog piece whenever you see the report quoted. Better still, read the report for yourself and make up your own mind.