SMotW #64: patching policy compliance

Security Metric of the Week #64: patching policy compliance

The idea behind this metric is to compare and reconcile the actual software patching status of corporate IT systems against the corporate policies and procedures on patching and vulnerability management.  

Clearly the details of the comparison and reconciliation depend largely on precisely what the policies and procedures demand, while the assessor (metricator) may be somewhat selective in assessing compliance.  So-called vulnerability assessment tools, for instance, typically search systems for installed software, determine the versions installed, then look up a database of known latest versions to see whether the software is up to date.  The process is almost entirely automated making it quite cheap and easy to run ... but Acme's corporate policies and procedures require rather more than just "Always install the latest versions of software", such as:

• Acme must maintain a database of installed software on all corporate IT systems;
• Newly-released patches for software used by Acme must be assessed to determine whether they are applicable and necessary (e.g. the issues they address are causing problems for Acme, or are risks of concern to Acme);
• Patches addressing security risks that are being actively exploited should be prioritized over those that are merely theoretical;
• Patches addressing vulnerabilities that are exposed to the Internet or other threat groups (including internal threats and business partners) should be prioritized over those that are not so exposed;
• Patches addressing security risks to business- or safety-critical systems should be prioritized over relatively low-risk systems;
• Applicable, necessary patches must be checked and if appropriate tested to ensure that they will not adversely affect the operation of Acme systems, especially business-critical systems - unless the Information Security Manager determines that the risk of not applying a critical security patch outweighs the risk of it causing operational problems;
• Patches must be reversible, in other words backups and other arrangements must be made to enable patches to be reversed-out (uninstalled) efficiently if problems appear in operation, even if the pre-installation checks and tests gave acceptable results.

Assessing compliance with a policy as complex as that requires a lot more effort than just running a vulnerability assessment tool.   

The patching policy compliance status has some significance and Relevance to security, since systems that are not being properly patched probably expose a number of technical vulnerabilities due to design flaws and bugs in software.  Furthermore, the patching status may be somewhat indicative of policy compliance in general, meaning that this metric could be used as a compliance indicator.  However, it is glaringly obvious from the PRAGMATIC score of just 37% that the metric is not favored by Acme management.  In their assessment, it is unlikely to be Timely (time spent assessing compliance might be better spent patching).  It is cumbersome (i.e. Costly) to measure, and is measured by people with a direct interest in the subject areas (i.e. it lacks Independence or integrity).

P	R	A	G	M	A	T	I	C	Score
66	52	55	77	19	36	11	8	5	37%

The low score for Meaning is interesting.  Usually, a low score here indicates that the metric isn't self-evident and would need to be explained to the intended audience or risk them misinterpreting it.  In their assessment, Acme managers considered that the metric would primarily be an operational level metric, aimed at information security and IT professionals.  As such, the fact that the metric is quite technical in nature would not be an issue since the intended technical audience can be expected to understand what it means.  However, in this case, the metric does not provide sufficient technical detail to be useful.  In particular, as currently worded, it does not indicate any kind of priority for action.  Which systems need to be patched first, and why?