Asking wise questions

"You can tell whether a man is clever by his answers.

You can tell whether a man is wise by his questions."

Naguib Mahfouz

Posing good questions is one of the things that research scientists learn to appreciate early on: scoping and framing an issue in such a way that the questions which arise can actually be answered through feasible research is important, but there's more to it than that. Although narrowly-focused, specific questions tend to be easier to answer, the answers are usually just as narrow and specific. Whereas broader, deeper, bolder questions relating to bigger concerns are much harder to deal with and answer, they can generate greater insight.

"How many spam emails were wrongly classified by the anti-spam software as ham last month?" is an example of a narrowly-scoped information security question, relatively straightforward to answer ... but of little real consequence to the business. The number that is generated by this metric - the measurement - may be factually correct and quite precise, but what does it actually mean? It may have more value if it were expressed relative to prior months, highlighting the trend, but even then it would most likely prompt, say, the CEO to ask "So what?". We have almost certainly posed the wrong question if the CEO doesn't even care whether it is answered, especially given that there are many other much bigger issues on the top table.

The number of wrongly classified spams is clearly a poor security metric for the management audience, although it may perhaps be useful for, say, the technician fine-tuning the anti-spam parameters.

Odd, then, that much of the present-day discussion around security metrics concerns narrow technical metrics just like that. Over in some dark and dusty corner of the Internet, a gaggle of security metrics people and vendors are excitedly chatting about technical vulnerability and patching metrics. They are fine-tuning their measures of packets dropped by the firewalls and malware trapped by their antivirus solutions. 

"So what?" doesn't even seem to have occurred to them, as yet.

If instead, we simply re-framed the question along the lines of "How much did spam-emails-wrongly-classified-as-ham cost us last month?" the answer would have more obvious meaning and relevance to the business. While the measurement and analysis would take a bit more work, expressing the answer as a dollar figure would instantly provides a clue, a way for the audience to assess the scale of the spam issue relative to various other business matters. Even if a business manager was not entirely certain about the meaning of spam and ham, he/she would instantly appreciate that $3m is a materially different matter to $300, whereas who cares if there were three hundred or three million wrongly classified spams? There's no scale, no point of reference. 

Some of you may be thinking "Aha! Let's simply express our management-level information security metrics in dollars!" but financial metrics are not the be-all-and-end-all. Dollars make sense, true, and dollars are important, but governing and managing the corporation well goes beyond purely financial considerations. The CEO is unlikely to be fascinated by the dollar cost of wrongly-classified spam, packets dropped or viruses blocked, unless the costs are huge anyway, in which case he/she is probably already painfully aware of the issue!  

By the way, I deliberately just used the word 'fascinated'. Metrics that are plain boring have little if any impact, and so might as well not exist. Metrics that are exciting, that interest and engage the audience, prompting them to ask related follow-up questions such as "Why?" and "What should we do about this?", are the ones many of us are seeking. This is why PRAGMATIC includes the Meaningfulness criterion. Effective metrics provide information that makes sense, resonates with and ultimately motivates or fires-up the audience. "So what?" shouldn't even come into it.

The more general solution is to pose the questions that matter, which means framing and understanding the issues that matter, which in turn begs the question "What matters?". In chapter 2 of our book, we wrote at some length about the purpose of metrics, why things are measured, including this little tip:

"This is a deceptively important chapter, more than just a way to introduce the book. While you ponder the questions we raise here, ask yourself other similar questions that are relevant to you and your organization, your management, your information security situation. Better yet, consider the way in which we have phrased the questions as, in so doing, we are subtly framing the problem space. Posing appropriate questions is the real art to metrics. Answering them is relatively easy! If you gain nothing else from this book, we sincerely hope you pick up some hints on framing and posing better questions of information security."

We continued by posing a handful of tough, big-picture questions such as "Are we secure enough?" and "Are we sufficiently compliant?", and then exploring several other key reasons for measuring information security, such as to improve information security, for strategic, tactical and operational reasons, for compliance and assurance purposes and so on. These are the whats in "So what?", the undeniable, straightforward, obvious reasons or purposes for which we need security-related measurements.

The strategic perspective is particularly important for security metrics since it determines the organization's entire approach to information security. Senior management support for, and investment in, information security depends on their understanding and appreciation of the risks and opportunities in that sphere, relative to other aspects of the business. Strategic high-level security metrics give meaning to the budget requests and substance to the business cases. From a governance perspective, management cannot knowingly disregard such serious matters (although there may be things they would rather not have been told!). Get the strategic security metrics right and the rest will follow.

Against that backdrop, what is the point in measuring and reporting the number or even the dollar value of misclassified spams?

Take a long hard look at chapter 2. Think about it in the context of your organization - your strategic concerns, your management's pain-points, the business objectives for information security and risk management.  Antivirus, antispam and firewalls are merely a means to an end: that 'end' is the important bit. Do your security metrics address the right questions, the questions that matter?

Rgds,

Gary

[PS  The issue of 'asking the right questions' goes way beyond information security metrics. Here in New Zealand, for instance, a Citizens Initiated Referendum is currently asking:

"Do you support the Government selling up to 49% of Meridian Energy, Mighty River Power, Genesis Power, Solid Energy and Air New Zealand?" 

I have no idea which citizens chose the question nor how they did so, but it smells distinctly like the product of a committee, no doubt one that has been vociferously discussed and argued over, endlessly word-smithed, tweaked and compromised. Aside from being wordy, the question is ambiguous and confusing. Is it one question, in fact, or five rolled into one? Several terms in the question are over-laden with meanings and implications. Is it getting at a party political issue about the government in power, or about what's best for the nation? Does "support" mean "agree with" or "accept"? And beneath it all lurk much bigger but unstated questions relating to the distinction between the Government's and the citizens' assets, state ownership, liabilities and capitalism. As a strategic metric for the government and the country, it's not exactly PRAGMATIC ... and if the politicians had a say in its formulation, perhaps that suits their purposes, along with the predicted low turnout. The asset sales have already started. The strategy is already in play. A contrary referendum result would be, errrr, 'unfortunate'.]

[PPS  I might be clever enough to deal with incidents, but it probably would have been wise to avoid them!]