SMotW#13: unauthorized software

Security Metric of the Week #13: proportion of purchased software that is unauthorized

This week's metric is based on a suggestion from ISACA.  A large number of metrics are either directly suggested or at least implied in various ISACA papers. It doesn't particularly matter where this specific metric was suggested, and in any case we've almost certainly departed from ISACA's original wording, but that's really not the point: the fact is that there are loads of possible metrics out there.  ISACA is but one of many sources of inspiration.  We've plucked this metric out of thin air simply as a vehicle to demonstrate how the PRAGMATIC method is applied.  

The metric's title is somewhat ambiguous, so we start by elaborating on it to figure out what we think it might have been intended to measure.  In referring to the purchase of software and authorization, it seems to be focusing on the procurement processes, but who knows?  Well the author of the original suggestion would undoubtedly have had a particular situation in mind, perhaps a specific control weakness that he/she had identified in the course of IT audit work.  Perhaps the metric was suggested as a result of the prevalence of unlicensed (pirated) software in the average corporation, or maybe it is hinting at software downloaded from the Internet and used by employees without any form of management authorization or indeed procurement process.  

If we were to actually use this metric as originally defined, there are immediate questions about its true Meaning and Relevance.  The Meaning issue could be addressed by clarifying the metric and laying out a more detailed specification.  However, its Relevance to information security is not so easily resolved.  Is it in fact meant to identify issues with the company's procurement or software installation processes, or to point out the information security consequences that might arise?  Either aspect might be important to management but we'd best decide which one we are trying to measure.

Measuring the metric would presumably involve checking through the company's purchasing records in order to reconcile software purchases against some form of ‘authority to purchase’, such as duly authorized purchase orders.    We might also have to reconcile inventory records of software installed on corporate systems against procurement records to find software that has mysteriously appeared on the network without being formally procured and authorized.  And for good measure (pun intended!), we really ought to check that the software inventory accurately reflects the software actually installed, which means auditing the software on at least a reasonable sample of corporate systems in detail.  The upshot of all that laborious, tedious work is, of course, a significant Cost.  There are automated tools that can create inventories of installed software, but the best packages aren't free and there would be substantial costs involved in configuring and running the checks and analyzing the data, especially if it was intended to repeat the process regularly.  In short, Cost is bound to be a major drawback to this metric, especially given doubts over its benefits (the Cost criterion is more accurately described as Cost-effectiveness or net value).  

As the analysis continues and the PRAGMATIC score gradually comes together, further potential issues some to light with this candidate metric.  For instance, IT are the people most likely to be asked to measure the metric since they have the necessary access to most if not all the corporate IT systems.  However, IT are also the people most likely to be in the hot seat for creating or allowing unauthorized software to be installed, creating a possible conflict of interest and calling into question their Independence.  This issue might be addressed by using IT auditors, but doing so would further increase the metric's Costs, and slow down the measurement process impacting the metric's Timeliness, already an issue of concern.

In the imagined context of Acme Enterprises Inc., the PRAGMATIC scoring table for the metric ends up something like this:

P	R	A	G	M	A	T	I	C	Score
71	51	90	75	82	35	13	20	6	49%

If you were undertaking this analysis back on the ranch, you would probably score the metric differently in various respects for rational reasons that make sense in the context of your organization.  Perhaps your management is struggling with particular software authorization issues that are directly Relevant to this metric.  There may well be counter-arguments and other factors to take into consideration, and in the course of analyzing and discussing the scores, you will quite likely think of ways in which the suggested metric might be adapted somewhat to make it more useful and in so doing improve its PRAGMATIC score.  Congratulations: you are using the process pragmatically!