Metrics to govern and manage information security

Section 9.1 of ISO/IEC 27001:2013 requires organizations to 'evaluate the information security performance and the effectiveness of the information security management system'.  The standard doesn't specify precisely what is meant by 'information security performance' and '[information security?] effectiveness' but it gives some strong hints:

"The organization shall determine:

a) what needs to be monitored and measured, including information security processes and controls;

b) the methods for monitoring, measurement, analysis and evaluation, as applicable, to ensure valid results;

c) when the monitoring and measuring shall be performed;

d) who shall monitor and measure;

e) when the results from monitoring and measurement shall be analysed and evaluated; and

f) who shall analyse and evaluate these results."

The standard specifies (much of) the measurement process without stating what to measure i.e. which metrics.  No doubt the committee would argue that it is not possible to be specific about the metrics since each organization is different - and there's a lot of truth in that - but it's a shame they didn't explain how to select metrics or offer a few examples ... which is where our security awareness paper originally delivered in August 2008 picks up the pieces.

We drew on the IT Governance Institute's advice on information security governance for inspiration, suggesting metrics corresponding to the four aspects identified in the ITGI paper (governance outcomes; knowledge & protection of information assets; governance benefits; and process integration).

[The original hyperlink to the ITGI paper now gives a 404 page-not-found error, unfortunately.  It was a good paper.  Perhaps they moved or updated it?]

Infosec & risk management metrics

We've just republished the next in the series of management-level security awareness papers on metrics.  The latest one lays out a range of metrics for information security and risk management.

Leaving aside the conventional metrics that are typically used to manage any corporate function, the paper describes those that are peculiar to the management of information risk and information security, with an emphasis on business-focused metrics.

I spent last week teaching a CISM course for ALC in Sydney.  The business and risk focus is a unifying thread throughout CISM, from the governance and strategy angle through risk and security management to incident management.

In contrast to courses covering the more technical/IT aspects of information security intended for mid- to low-level information security professionals with operational responsibilities, CISM is intended for Information Security Managers and Chief Information Security Officers with governance, strategic and management responsibilities.  It promotes the value of elaborating on business objectives that are relevant to information risk and security management, and using those to drive the development and delivery of a coherent business-aligned risk-driven information security strategy.  Metrics are of course integral to the CISM approach, particularly governance and management metrics similar to those in the awareness paper.

Awareness paper on authentication and phishing metrics

We've just republished a management-level security awareness paper on metrics relating to user authentication and phishing.

The introduction asks "How do we tell whether our authentication controls are effective?" and "What does 'effective' even mean in this context?" - two decent questions that could be addressed through suitable metrics.

Questions like these are central to the GQM (goal-question-metric) method (see IT Security Metrics by Lance Hayden), and not just literally in terms of their position in the handy acronym. They link the organization's goals or objectives relating to information security, to the information security metrics that are worth measuring.

In your particular circumstances, the effectiveness of authentication controls might or might not be of sufficient concern to warrant generating the associated metrics. Other aspects might take precedence, for example the amount invested in authentication controls, and the ongoing operating and maintenance costs of those controls. It's usually not too hard to think up a whole raft of aspects, parameters or concerns relating to the topic area, but focusing on the things that are likely to matter most to the organization (business priorities) is a good way to keep the list within reasonable bounds. Once you know what they are, the next step is to figure out the questions arising e.g. "Are we spending appropriately (neither too much nor too little) on authentication?"

From there, it's simply a matter of deciding what data would help address the questions, and those are your metrics!  Job done!  Errr, well, no, not quite: if you have several goals/areas of concern and numerous questions arising, each requiring multiple metrics to generate the answers, there is a distinct risk of being overwhelmed with possibilities. It is infeasible and in fact counterproductive to attempt to measure everything. Less is more! This is where the PRAGMATIC method comes into play as a way to whittle down the long list to a shortlist of metrics showing the most promise. The GQM approach also suggests filtering out the metrics that don't address the questions very well, and trimming down on metrics addressing questions that are only marginally related to the organization's business goals. Both approaches have their merits.

3 more metrics papers

We've just published another three documents on security metrics, written and first released five years ago as part of the management stream in the NoticeBored information security awareness service.

The first paper was concerned with measuring integrity.  Despite being one of the three central pillars of information security, integrity is largely overshadowed by availability and, especially, confidentiality ... and yet, if you interpret 'integrity' liberally, it includes some extremely important information security issues. The 'completeness and correctness' angle is pretty obvious, while 'up to date-ness' and 'appropriateness' are less well appreciated.  Add in the character and trustworthiness of people, and integrity takes on a rather different slant (Bradley Manning, Julian Assange and Edward Snowden springing instantly to mind as integrity failures).  An 'honesty metric' is an innovative idea.

The integrity metrics paper also suggests measuring the integrity of the organization's security metrics program or system of measurements, on the basis that metrics ought to be accurate, complete, up-to-date and relevant. The metrics integrity issue is obvious when you think about it. Managing with poor quality information is less than ideal.  However, in our experience, information security metrics are mostly taken at face value: we usually focus on what the numbers are telling us without even considering that they might perhaps be wrong, misleading, incomplete or inconsequential. Worse still, we get so distracted by the fancy "infographics" that the information content is almost irrelevant.  That's hardly a scientific approach!  We have raised this issue before in relation to treating published security surveys as gospel, blythely ignoring the fact that most are statistically dubious if not patently biased marketing copy. Remember this the next time you search the web for pie charts to illustrate your security investment proposals, or the next time someone tries to persuade you to loosen the purse strings! 

A short, humdrum paper on IT audit metrics suggests a few ways to measure the IT audit function, such as "IT audit program coverage" as well as conventional management metrics.  

The third paper on malware metrics was virtually the same as the version released a year earlier. We made some changes the following year, partly due to the research and thinking that went into writing PRAGMATIC Security Metrics ... but you'll have to wait just a bit longer for the 2009 paper.

Management awareness paper on contingency metrics

Here's the next security awareness paper in the series, describing metrics relating to contingency and business continuity management.

"Measuring the effectiveness of contingency arrangements is a tough challenge, not least because (like insurance policies) we hope we will never need to use them. However it makes sense to measure our investment in contingency plans and preparations, and to confirm whether management is sufficiently confident in them, prior to enacting them as by that stage it will be too late."

Possible contingency metrics suggested in the paper include:

• RTO and RPO - classic disaster recovery metrics in their own right
• Resilience - measured by incidents
• Recovery - proportions of systems for which RTP/RPO are defined, tested and met
• Costs - easier to measure than benefits, and yet an uncommon metric in practice
• Management confidence - to what extent do managers believe in the congtingency arrangements?

There are many other possible metrics in this area.  What do you measure? Why?  What do your contingency or business continuity metrics tell you?  

Management awareness paper on office information security metrics

The NoticeBored security awareness module from which we've plucked this management-level discussion paper covered information security issues relevant to the typical office or corporate workplace.

In effect, offices are information factories. Office information security controls are essential to keep the factory, its machine tools, operators and production processes running smoothly, efficiently and profitably, and to protect office-based and accessible information assets (paperwork, computer files, and white-collar workers) from all manner of risks.

Office security concerns include:

• Intruders - burglars, industrial spies and 'lost' visitors wandering loose about the place
• Fires, floods and accidents 
• Various logical/IT security incidents affecting the office network and file system, workstations, email and other applications
• Procedural issues such as workers' and visitors' failure to comply with office information security policies and procedures.

This short awareness paper outlined just a few office security metrics, without delving into details. At the time it was written (2008), we lacked the means to analyze metrics in much detail since the PRAGMATIC approach had not yet been invented. Looking back on it now, the paper is fairly typical of its day, quite naive in approach, leaving the reader to contemplate and perhaps choose between the metrics suggested.  

Management awareness paper on social engineering metrics

Security awareness is the primary control against social engineering, hence this is an essential core topic for the awareness program. Making managers aware of how they might measure [the risks and controls relating to] social engineering is the purpose of this awareness paper.

The paper illustrates how elaborating on the control objectives helps to identify relevant security metrics. For example, the objective to 'make the entire workforce aware of social engineering' suggests the need to measure the security awareness program's coverage. 

The paper identifies just three security awareness metrics. There is nothing special about those particular metrics, and they are certainly not the only ways to measure awareness. It is deliberately left as an exercise for the reader to determine firstly whether it might indeed be worth measuring coverage of the awareness program, and if so secondly how best to do that.

By the way, in conjunction with fellow author Walt Williams, I'm currently developing a new information security awareness maturity metric in the same style as the maturity metrics in the book. It should be ready to publish later this month. Watch this space!

Kind regards,
Gary 

Management awareness paper on security compliance metrics

Compliance with information security related obligations, privacy laws in particular, was already a major issue for management when this paper was written back in 2007. Over the succeeding years, it has grown even bigger and yet we still often hear people discussing compliance in simplistic, black-and-white or binary terms in the sense of "You either comply or you don't". In reality, compliance is usually a matter of interpreting and weighing-up the evidence concerning the extent to which the obligations have or have not been fulfilled, and their relative importance. Compliance may not be glorious Technicolor but there are definitely shades of grey!

This metrics briefing proposed a few simple measures of the extent and speed of compliance, as well as the costs relating to or arising from compliance.  

In addition to legislation, it mentioned compliance with and enforcement of corporate policies and other requirements (such as good security practices and contractual obligations - PCI-DSS being a classic example).  

We developed further and elaborated on the concept of a 'security compliance status' metric that was introduced in this paper in later briefings. Looking at the paper now, with the benefit of hindsight, it seems rather naive but it served a purpose as a security awareness item for managers.

Management awareness paper on physical security metrics

In the context of information security, physical security is about protecting tangible assets holding, communicating or processing valuable information - primarily ICT systems and data storage media - from physical incidents such as theft, criminal or accidental damage, loss, sabotage, fire, flood, mechanical breakdown, electrical surges, dips and power cuts, static discharge, magnetic or electrical interference etc. that would damage the information content or the services provided.

Strictly speaking, it includes physical protection for people, workers particularly, since we also constitute physical information assets - well most of us anyway (some are liabilities!).  'Health and safety' is, in a sense, part of information security, along with substantial parts of HR.

This very brief metrics discussion paper, written seven years ago, does not explore the entire scope of physical security but mentions just a few considerations around physical security targets and measurements.  It was not one of our best efforts ... and yet it might just prompt you to think of something worth measuring in your situation.

I promise the quality of this series of papers improves as we head into 2015. Our understanding of metrics improved markedly as we did the thinking and research for the PRAGMATIC book, on top of which we revisited, updated and expanded on the older papers as we completed successive cycles of information security topics. Yes, I know it's "jam tomorrow" but stick with us and enjoy the journey. 

Management awareness paper on email security metrics

Measuring the information security aspects of email and indeed other forms of person-to-person messaging implies first of all that you understand what your security arrangements are intended to achieve.  What does it mean to "secure email"?  If that's too hard to answer, turn it on its head: what might be the consequences of failing adequately to secure email? Does that help?

Our next metrics discussion paper opens with a brief analysis of the 'requirements and targets', also known as the objectives, of email security, expressed in broad terms. For instance, preventing or at least reducing the issues relating to or arising from spam and malware is a common objective ... hence one might want to measure spam and email-borne malware, among other aspects. 

That in turn begs questions about which specific parameters to measure and how - for instance, there are many possible ways to measure spam, such as the:

• Number of spam emails arriving at the organization, or rather the rate of arrival (spams per hour, day, month or whatever);
• Number of spam emails leaving the organization (!);
• Types of spams detected, perhaps analyzed according to the differing threats they represent, ranging perhaps from trivial/timewasting to targeted spear-phishing attacks;
• Proportion of spam that is detected and blocked versus that passed to users (and, hopefully, identified by them as spam);
• Cost of anti-spam controls, including the anti-spam software and systems, network and system load, user and support time spent on the problem etc.

For each of those potentially interesting concerns, there may be several actual ways to generate the corresponding measures, and several ways to analyze, present and hopefully use the data. The paper outlines just a few examples to illustrate the approach, but it should be obvious from the above that we could easily have written a very long and boring treatise just on email security metrics. We didn't ... because the paper was 'just' written for awareness purposes: it is designed to get managers thinking and talking about security metrics, opening their eyes to the possibilities and encouraging them to consider their own, specific information needs as opposed to the few generic examples given. Rather than boring the pants off our readers, we're hoping to intrigue and stimulate them, catch their imagination not write a book on it.  [By the way, that's one of the objectives for the security awareness program, implying that perhaps we might measure it in order to confirm whether the awareness program is achieving the objective, and if not suggest how we might do better!]

In case you missed it, there's an important point here with implications for all metrics (not just infosec metrics!). What we choose to measure depends on our information needs, organizational circumstances, objectives, challenges, maturity level and so on.  Your situation is different, hence you will probably be better served by a different set of metrics. Much as we would like to offer you a little pre-canned set of email security metrics, chances are they won't work terribly well for you. They'll be sub-optimal, perhaps costly, distracting and generally unhelpful. Remember this when anyone enthuses about particular security metrics, or when someone naively posses the classic "What metrics are best for X?"

One might argue that since we share a number of common information security challenges (such as spam), we might benefit from a number of common metrics ... but that's overly simplistic, just as it would be nuts to suggest that we should all employ identical anti-spam controls. Some email security metrics might well turn out to be more or less common than others but that alone is not a sensible reason to select or avoid them respectively.

This is why we invented the PRAGMATIC approach. While it is easy to come up with long lists of possible metrics using methods such as Goal-Question-Metric (see Lance Hayden's book "IT Security Metrics") and metrics catalogs (e.g. the 'consensus security metrics' from CIS), there was a distinct lack of guidance on how to shortlist and finally choose the few metrics actually worth implementing.

Management awareness paper on trade secret metrics

Protecting proprietary information, especially trade secrets, is - or rather should be - a priority for almost all organizations.

Trade secrets can be totally devalued if they are disclosed to or stolen by competitors, if that leads to their being exploited. The loss of competitive advantage can decimate an organization's profitability and, in the worst case, threaten its survival.

Availability and integrity are also of concern for proprietary information. If the information is destroyed or lost, the organization can no longer use it. If it is damaged or corrupted, perhaps even deliberately manipulated, the organization might continue to use it but is unlikely to find it as valuable.

Significant information security risks associated with proprietary information imply the need for strong, reliable information security controls, which in turn implies the need to monitor the risks and controls proactively. Being just 3 pages long, the awareness paper barely introduces a few metrics that could be used to measure the organization's security arrangements for trade secrets in 5 or 6 categories: its purpose is not to specify individual metrics so much as to lift the covers on a few possibilities. Your organization needs to determine its own security measurement objectives, adopting metrics that suit your purposes. Hopefully, this brief security awareness paper will stimulate thought and discussion on metrics: where it leads from there is down to you.

Management awareness paper on authentication metrics

User identification and authentication (I&A) is a key information security control for all systems, even those that allow public access (unless the general public are supposed to be able to reconfigure the system at will!). As such, it is important to be sure that I&A is working properly, especially on business- or safety-critical systems, which in turn implies a whole bunch of things. I&A must be:

• Properly specified;
• Professionally designed;
• Thoroughly tested and proven;
• Correctly implemented and configured;
• Used!;
• Professionally managed and maintained;
• Routinely monitored.

Strangely, monitoring is often neglected for key controls. You'd think it was obvious that someone appropriate needs to keep a very close eye on the organization's key information security controls, since (by definition) the risk of key control failure is significant ... but no, many such controls are simply implemented and left to their own devices. Personally, I believe this is a serious blind spot in our profession.  

If unmonitored key controls fail, serious incidents occur unexpectedly. In contrast, management has the opportunity (hopefully!) to spot and respond to the warning signs for key controls that are being routinely monitored and reported on using suitable metrics.  Security metrics, then, are themselves key controls.

The management-level awareness briefing paper briefly sets the scene by outlining common requirements for I&A. It then briefly describes four types of metric that might be used to monitor, measure and generally keep an eye on various aspects of the control. Perhaps the most interesting is the authentication failure rate ... but to be honest my thinking on metrics has progressed in the 7+ years since this paper was written. The metrics in the paper look naive and basic to me now. Since I'm updating the authentication awareness module this month, I'll be thinking up better I&A metrics when I rewrite the paper for NoticeBored subscribers ... perhaps even scoring them using the PRAGMATIC method ... oh and revising those awful graphics!

Management awareness paper on insider threat metrics

How do you measure 'insider threats' in your organization?  

If your answer is "We don't!", then I have to wonder how you are managing insider threats. 

Without suitable metrics, how do you figure out how much of a problem you might have from employees, contractors, consultants, temps and interns?  How do you determine where best to spend your security budget? How do you persuade management to loosen the purse strings sufficiently to address the risks?  I guess you guess!

The NoticeBored discussion paper breaks down 'insider threat' into chunks that can be measured sensibly.  The main divide falls between deliberate attacks (such as frauds by insiders) and accidents (such as mistakenly overwriting the entire production database - don't laugh, it happened to me 25 years ago and the nightmare still haunts me today!). 

The paper picks up on one of the most productive sources of information security metrics: the IT Help/Service Desk's problem and incident management process.  Most mid-to-large organizations these days route employee queries and problem reports through a centralized helpdesk function that acts as a clearing house, offering first-line support and routing unresolved calls to the appropriate resolving agencies for specialist help.  A valuable core part of their role is to ask questions and record information about calls in the ticketing system, tracking the calls through to completion.  The system, then, is a goldmine of information about queries, problems, events, incidents and other issues, including "near misses" (assuming the organization is sufficiently clued-up to encourage workers to report close-shaves as well as actual incidents).

If this is all Greek to you, find a spare hour or two to speak to a helpdesk supervisor about the call-ticketing system, about how calls are categorized and recorded, and how to squeeze suitable reports out of the system ... but, before you get completely carried away on a wave of enthusiasm, planning how to use all that sexy statistical and textual information, think very carefully about what you are doing.  The availability of information is not, in fact, the main concern when designing security metrics.  A far more important issue is to figure out what you want to know, what questions need to be answered.  Conceiving questions that fit the available answers is putting the cart before the horse.

Management awareness paper on network security metrics

Measuring network security involves, first and foremost, determining what 'network security' encompasses, and how it relates to the business.

Writing way back in 2007, we said that network security "comprises a range of technical and procedural controls designed to prevent, detect and/or recover from security incidents affecting the corporate data networks – incidents such as unauthorized access (hacking), worms and other malware infections, and unplanned network downtime".  The context for the paper was a NoticeBored information security awareness module exploring security arrangements protecting data networks against both deliberate and accidental threats.

The paper described ways to measure network security incidents, controls, risks, compliance and governance.  It ended with an upbeat conclusion and call-to-action: "Do not neglect the value of having the experts present and discuss reports with management.  The dialogue that ensues adds value to the written reports.  Why not present and discuss these ideas with your management and seek their opinions, bringing to the table some prototype reports in one or more formats to stimulate discussion and clarify their objectives?"

The PRAGMATIC approach is a structured, systematic way to consider the pros and cons of various metrics, leading ultimately to a decision on which ones (if any!) to adopt.  Just as important as the final destination, the method leads managers and infosec pro's together on a journey of discovery.

My guess is that the network security incident and compliance metrics would probably score above the others proposed in the paper but I can't tell for sure because I don't know a thing about your situation or measurement needs: your evaluation may well come to a markedly different conclusion.  Furthermore, in discussing the metrics paper, you might come up with 'variations on a theme', meaning variants of the metrics proposed that would score more highly, and perhaps something completely different, especially if it turns out that none of the metrics in the paper are suitable.  

That spark of creativity is the real power of PRAGMATIC.  Scientifically analyzing the factors determining the strength and suitability of the metrics under discussion leads to a better understanding of the metrics and of the measurement needs.  Contrast that with the blank-sheet approach.  Faced with the question "What network security metrics shall we adopt?", most business managers would be at a loss to know how to proceed.  If the security, network or IT people suggest one or more technical security metrics to fill the void (perhaps plucked from the air or picked out of some banale list with a pin), the managers don't have the basis for evaluating them, meaning that they are quite likely to accept whatever is offered, perhaps later coming to realize that they aren't exactly ideal.  If the metrics truly don't work, it's back to the drawing board to pick some more ... assuming someone has the good sense to call a halt to the senseless waste of time and effort (that's not a facile comment: many organizations drift aimlessly along for years with inappropriate, unsuitable or low-value metrics because everyone assumes someone else must be using them!*).

Although it is possible for the organization to develop a decent set of metrics by sheer trial-and-error, there is a distinct chance that good metrics will be discarded or discounted for arbitrary reasons, and that opportunities to 'tweak' metrics the better to fit the business needs will be missed.  Using the PRAGMATIC method as a systematic process to select, develop and maintain suitable metrics has to be a better way, don't you think?

Kind regards,

Gary

* That reminds me: have you audited your organization's security metrics? It may be a tedious assignment but a competent and diligent auditor should be able to follow the paper trail from gathering the source data through analysis and reporting to the decisions arising - if any.  If the trail goes cold, that's a strong indication that the metrics are simply not working, implying not just a waste of effort and money but an information void and lost opportunity.  Low quality metrics are a costly distraction, literally worse than useless!

Management awareness paper on malware metrics

Malware - malicious software - encompasses a variety of computer viruses, Trojans, network worms, bots and other nasties.  

Malware has been the scourge of IT users ever since the Morris worm infected the early Internet way back in 1988.  Despite the enormous global investment over the intervening years in information security controls against malware (including security awareness!), it remains a significant security concern today. 

Although antivirus software companies sometimes admit that they are fighting a losing battle, malware is generating so much income both for the VXers (malware authors) and their criminal masterminds, plus the antivirus software companies, that the arms race looks set to continue for the forseeable future.  Both sides are constantly investing in new tricks and techniques, fuelling a thriving black market in zero-day exploits and novel malware.

Meanwhile, the rest of us are lumbered with paying for it in one way or another. Addressing the malware risk is more involved than simply throwing money at antivirus software: malware metrics are the key to understanding the balance of risk against control, investing wisely, and doing our level best to keep up with, if not stay one step ahead of the game in this dynamically evolving situation.

Download our third security metrics awareness paper here.

Management awareness paper on database security metrics

The next NoticeBored security awareness paper suggests to management a whole bunch of metrics that might be used to measure the security of the organization's database systems.

Most information-packed application systems are built around databases, making database security a significant concern for the corporation.  We're talking about the crown jewels, the bet-the-farm databases containing customer, product and process information, emails, contracts, trade secrets, personal data and so much more.  Despite the importance of database security, we don't know of any organization systematically measuring it ... although we do know of many that struggle to keep on top of database security design, development, testing, patching, administration and maintenance!

So how exactly are management supposed to manage database security without database security measures? Extra sensory perception, perhaps, or gut-feel? Either way, it's hardly what one might call scientific management!

Download the paper here.  We'd be fascinated in your thoughts.  Do any of these measures catch your imagination?  What other database security metrics or measurement approaches would you suggest?  What do you use?

Management awareness paper on IPR metrics

When we get a spare moment over forthcoming months, we plan to release a series of awareness papers describing metrics for a wide variety of information security topics through the SecurityMetametrics website.

The first paper, dating back to 2007, proposes a suite of information security management metrics relating specifically to the measurement of Intellectual Property Rights (IPR). Managing and ideally optimizing IPR-related controls (namely the activities needed to reduce the chances of being prosecuted by third parties for failing to comply with their copyright, patents, trademarks etc. plus those necessary to protect the organization's own IPR from abuse by others), requires management to monitor and measure them and so get a sense of the gap between present and required levels of control, apply corrective actions where necessary and improve performance going forward.

These metrics papers were originally delivered to subscribers of the NoticeBored security awareness service, as part of the management stream.  Their primary purpose is to raise awareness of the monthly topic, but really we hope to encourage information security professionals and management to think about, discuss and perhaps adopt better security metrics.  

If you follow the sequence, you'll notice our own thinking change over the 7 years since this first paper, particularly while PRAGMATIC Security Metrics was being written.  From time to time, we introduced new styles of metric, often covering the same information security topics repeatedly but from slighly different angles (there are currently 50 infosec topics in the NoticeBored portfolio, with still more to come).

If you'd like to discuss any of these papers, please comment here on the blog or through Google Plus.