Pre-PRAGMATIC

Hitherto - before the PRAGMATIC method was invented - deciding which security metrics to measure was a black art, a highly subjective decision making process.  One might even question whether organizations actually 'select' security metrics deliberately, systematically and rationally.  


Think about that for a moment.  Why does your organization measure whatever it does measure in relation to information security?  Does that mean that management doesn't care about all the other security stuff you could also measure?  Does it really matter what security metrics you use?


Pre-PRAGMATIC organizations presumably measure certain facets of information security because: 

• They are cheap and easy to report, typically because the raw numbers are readily available (some systems generate pretty graphs straight out of the box, but does management need them?);
• They are recommended by someone, peers claim to measure them, or the organization is required to report them by some third party (e.g. an authority such as a regulatory body or owner);
• 'It's the way we've always done it' ... (brains in neutral!);
• 'It seemed like a good idea at the time' ... although we may no longer recall why, and things have probably changed in the interim; 
• 'It's obvious!' ... but probe deeper and you may discover a mix of rational and irrational reasons for focusing attention on those particular aspects, and often a lack of appreciation of the opportunity cost i.e. it might be better to measure other things, or measure the same things in other ways;
• Management think they couldn't possibly measure the security things they really care about, and hence are forced to accept whatever metrics they are offered (nonsense!  Read Douglas Hubbard's book and get creative!).

Once an organization adopts the PRAGMATIC approach, selecting security metrics becomes a much more straightforward process.  The chosen metrics can be described and justified lucidly and convincingly, giving legitimate reasons for choosing them.  Other potential metrics can be assessed objectively in relation to the existing metrics suite, using the PRAGMATIC criteria as a framework for the decision-making process.  The PRAGMATIC approach lets management identify and weed-out low-scoring security metrics that aren't earning their keep, not just saving the associated metrics generation and reporting costs but focusing finite management attention on more valuable metrics.