SMotW #34: homogeneity

Security Metric of the Week #34: organizational and technical homogeneity

The degree of homogeneity (sameness) or heterogeneity (variation or variability) within the organization and its technologies affects its aggregated information security risks, in much the same way that monoculture and multiculture crops may face a differing risks from natural predators, parasites, adverse environmental conditions etc.  A particular mold that successfully attacks a certain cultivar of wheat, for example, may decimate a wheat field planted exclusively with that cultivar whereas it may not take hold, making little impact on a neighboring field planted with a mix of wheat cultivars differing in their susceptibility or resistance to the mold.  On the other hand, under ideal conditions, the monoculture crop may do exceptionally well (perhaps well enough to counteract the effects of the mold) where the mixed crop does averagely.  

Homogeneity of technologies, suppliers, contracts etc. increases an organization's exposure to  common threats - for example, serious security vulnerabilities in MS Windows may simultaneously impact the millions of organizations that rely on Microsoft's products.  On the other hand, homogeneity means standardization, lower complexity and ‘economies of scale’, generally generating substantial business benefits.  It is clearly in Microsoft's commercial interests to be seen to address serious security vulnerabilities in its products urgently, or risk mass defection of its customers (those who aren't entirely dependent, at least!).

The overall PRAGMATIC score for this candidate metric is mediocre:

P	R	A	G	M	A	T	I	C	Score
67	70	40	59	67	50	33	65	45	55%

The metric rates poorly on both Timeliness and Cost due to the difficulties of gathering and analyzing suitable data with any kind of precision.  However, a quick-and-dirty low-Accuracy assessment might be sufficient get this issue raised and discussed at the top table, which might actually be good enough (we're hinting at the measurement objective - an issue we have hardly mentioned in the blog but which is covered at length in the book).  The metric may perhaps be measured using scoring scales that we have discussed in several previous blog postings, for instance.

Sitting at 40%, the Actionability rating is also depressed for two distinct reasons: 

1. It is not entirely clear what constitutes an 'ideal' amount of homogeneity, since, as we have just said, there are pros and cons to it;
2. There are obvious practical constraints on management's ability to change the organization's homogeneity even if they wanted to do so.  Senior management might institute a supplier diversity policy, for instance, but there is likely to be considerable inertia due to the existing portfolio of suppliers currently contracted.  In many cases, there will be overriding commercial or technical reasons to retain the current suppliers, on top of the natural affinity that emerges through social interaction between individual employees and their supplier contacts.

Bottom line: this candidate metric is unlikely to make the grade for Acme Enterprises Inc., but it may be valuable elsewhere.

Newton's take on security metrics

He may not have considered this at the time, but Sir Isaac Newton's three laws of motion are applicable to security metrics ... 

Law 1.  Every object in a state of uniform motion tends to remain in that state of motion unless an external force is applied to it. 

An organization lacking effective metrics has no real impetus to change its approach to information security.  Management doesn't know how secure or insecure it is, nor whether security is "sufficient", and has no rational basis for allocating resources to security, nor for spending the budget on security activities that generate the most value.  Hence, they carry on doing pretty much what they've always done.  They approve the security budget on the basis of "last year's figure, plus or minus a bit".  They do security compliance activities under sufferance, and at the last possible moment.  

The law of inertia is particularly obvious in the case of large bodies that continue to blunder through situations that smaller, more nimble and responsive ones avoid.  We're not going to name names here: simply check the blogosphere and news media for plenty of unfortunate examples of sizable, generally bureaucratic, often governmental organizations that continue to experience security incident after incident after incident.  Management shrugs off adverse audit reports, inquiries and court cases as if it's not their fault.  "Our hands are tied", they bleat, "don't blame us!" and messrs Sarbanes and Oxley groan. 

By the same token, the auditors, investigators, courts and other stakeholders lack the data to state, definitively, that "You are way behind on X, and totally inadequate on Y".  They know things are Not Quite Right, but they're not entirely sure what or why.  Furthermore, those who mandate various security laws, regulations and edicts have only the vaguest notion about what's truly important, and what would have the greatest effect.  Mostly they're guessing too.

Law 2.  The relationship between an object's mass m, its acceleration a, and the applied force F is F = ma. 

Applying a force to an object accelerates or decelerates it.  The amount of acceleration/deceleration is proportional to the force applied and the mass of the object.  Do we honestly need to spell out how eloquently this describes metrics?  For those of you who whispered "Yes!" we'll simply mention the concepts of proportional control and feedback.  Nuff said.

Law 3.  For every action there is an equal and opposite reaction.

An interesting one, this.  

Once organizations are designing, developing, selecting, implementing, using, managing and improving their suites of PRAGMATIC information security metrics, they will inevitably start using the metrics to make changes that systematically and measurably improve their security.  That's the action part.  

Newton might predict a reaction: what would that be?  

Well, one reaction will involve the human threats such as hackers, malware authors, fraudsters, spies and so forth: they will up their game in order to continue successfully exploiting those victims who are more secure, or of course direct their evil attentions to less secure victims, including those who lack security metrics and hence presumably still manage, direct and resource security using guesswork, gut feel, magic incantations, lucky charms and astrology.   "I've heard on the golf course|read in the in-flight magazine|been told by a little bird that competitor X only spends 5% of its IT budget on security.  Clearly, we're spending far too much!"

Another reaction will involve other parts of the organization - other departments who notice that, for once, information security has management's ear.  They are successfully justifying the security budgets and investments that they themselves would love to have.  Some will react negatively, challenging and undermining the security metrics out of jealousy and a desire to go back to the good old days (law 1 in action), while others will seize the opportunity to reevaluate their own metrics, finding their own PRAGMATIC set.

Yet another reaction will come from the authorities, owners and other stakeholders who can't help but notice the marked contrast between PRAGMATIC and non-PRAGMATIC organizations.  The former give them fact-based, reliable and most of all useful information about their information security status and objectives, while the latter mysteriously hint at celestial bodies and rabbits' feet.  We confidently predict that security compliance obligations imposed on organizations will increasingly specify PRAGMATIC metrics, and indeed the PRAGMATIC approach, as part of the deal.

Let's be realistic about it: the change will undoubtedly be incremental and subtle at first, starting with the thought leaders and innovators who grasp PRAGMATIC and make it so.  Gradually, the language of security metrics will change as the early adopters enthuse about their new-found abilities to manage security more rationally and scientifically than has been possible before, and others come to appreciate that at last they can make sense of the metrics mumbo-jumbo spouted by the consultants and standards.  The laggards who cling to their existing approaches like a drowning man clings to a sodden log will face extinction through increasing security threats and incidents, and increasingly strident pressure from their stakeholders to "be honest about security".

SMotW #33: thud factor

Security Metric of the Week #33: thud factor, policy verbosity index, waffle-o-meter

If you printed out all your security policies, standards, procedures and guidelines, piled them up in a heap on the table and gently nudged it off the edge, how much of a thud would it make?  

'Thud factor' is decidedly tongue-in-cheek but there is a point to it.  The premise for his metric is that an organization can have too much security policy material as well as too little.  Excessively lengthy, verbose, confusing and/or overlapping policies are less likely to be read, understood and complied-with, while compliance and enforcement would also be of concern for  excessively succinct, narrow and ambiguous policies. 

A scientist might literally measure the thud using an audio sound level meter, dropping the materials (stacked/arranged in a standard way) from a standard height (such as one metre) onto a standard surface (such the concrete slab of the laboratory floor), getting a sound pressure reading in decibels.  A diligent scientist would take numerous readings including controls to account for the background noise levels in the lab (he/she might dream of having a soundproof anechoic chamber for this experiment, but might settle for a series of experimental runs in the dead of night), checking the variance to confirm whether everything was under control ...

... but that's not really what we had in mind.  We were thinking of something far more crude, such as a questionnaire/survey using a simple 5 point Likert scale:

1. Silence, similar to a pin drop.
2. A slight flutter of papers.
3. A gentle jolt as the heap hits the floor.
4. A distinct thud.
5. A bang loud enough to make people turn and look.

More likely, we'd opt for a continuous percentage scoring scale using those five waypoints to orient respondents but allowing them to navigate (interpolate) between them if they wish.

At the high end of the scale, there is so much policy stuff that it has become a total nightmare in practice to manage, use and maintain.  Management keeps on issuing policies in a vain attempt to cover every conceivable situation, while employees appear to keep on taking advantage of situations that don't yet have explicit policies.  Worse still, issued policies are constantly violated because due to lack of awareness or confusion over them, caused in part by inconsistencies and errors in the policy materials.  Some policies are probably so old they predate the abacus, while others use such stilted and archaic language that a high court judge would be flummoxed.  There are policies about policies, different policies covering the same areas (with conflicting requirements, of course) and probably turf wars over who should be writing, mandating, issuing and complying with the policies.  If anyone does anything remotely unacceptable, security wise, there is probably a policy statement somewhere that covers it ... but unfortunately there is also probably another one that could be interpreted to sanction it.

For organizations right at the low end of the scale, security policies are conspicuous by their absence.  There may perhaps be some grand all-encompassing statement along the lines of a Vogon admonition:

"Information shall be secured." 

... but no explanation or supporting detail - no practical guidance for the poor sods who are supposed to be complying with it.  Consequently, people make it up as they go along, some of them naturally tending towards the "Do nothing" and "It's not my problem" approach, others believing that security requires absolutely anything and everything that is not explicitly required for legitimate, stated reasons to be blocked.  On the upside, periodic policy maintenance is a breeze since there is next to nothing to review and confirm, but what little material there is is so ambiguous or vacuous that nobody is quite sure what it means, or what it is intending to achieve.  Compliance is a joke: there is no point trying to hold anyone to anything since there are policy gaps wide enough to steer an entire planetary system through.  Management resorts to trite phrases such as "We trust our people to do the right thing", as if that excuses their appalling lack of governance.

There is a happy medium between these extremes, although it would be tricky to set a hard and fast rule determining the sweet spot since it is context-dependent.  It usually makes sense to have the security policies match those covering other areas in the organization (such as finance, HR, operations, governance and compliance) in terms of quality (taking account of aspects such as depth, breadth, integrity, utility, readability etc.), but on the other hand if those other policies are generally accepted as being poor and ineffective, the security stuff should be better, good enough perhaps to show them The Way.

In metrics terms, the subjectivity of the measure is an issue: thud factor is in the eye of the beholder.  One person might think there are "far too many bloody policies!" while another might say "You can never have enough policies - and indeed we don't."  Nevertheless, explaining the issue and persuading several suitable people to rate thud factor on a common scale is one way to generate objective/scientific data from such a subjective matter.  Imagine, for instance, that you really did circulate a survey using the 5 point thud factor scale shown above, and collected responses from, say, 20 managers and 30 staff.  Imagine the mean score was 2.7: that is close to the middle of the scale, which indicates the subjective opinion that there are 'about enough' security policies etc., meaning there is probably no burning need to create or destroy policies.  However, if at the same time the variance was 1.8, that would indicate quite a wide diversity of opinions, some people believing there are too few policies and others believing there are too many - in other words, there is limited consensus on this issue, which might be worth pursuing (especially as there is probably some confusion about the measure!).  If you had the foresight to encourage people to submit written comments while completing the survey, you would have a wealth of additional information (non-numeric metrics, if there is such a beast) concerning the reasoning behind the scores and, perhaps, some specific improvement suggestions to work on.  

Anyway, let's see how it scores as a PRAGMATIC metric in the imaginary situation of Acme Enterprises Inc.:

P	R	A	G	M	A	T	I	C	Score
82	80	60	60	70	45	85	86	84	72%

72% puts this metric surprisingly high on the list of candidates.  It turns out to be a potentially valuable security metric that might have been dismissed out of hand without the benefit of the PRAGMATIC analysis.

The PRAGMATIC method gives us more than just a crude GO/NO-GO decision and an overall score.  Looking at the specific ratings, we see that Accuracy is definitely of concern with a rating of 45%.  If Acme's management showed sufficient concerned at the policy quality issue and was seriously considering adopting this metric, there are things we could do to improve its Accuracy - albeit without resorting to nocturnal scientists!  For example, we might revise the wording of the Likert scale/waypoints noted above to be more explicit and less ambiguous.  We could be more careful about the survey technique such as sample sizes and statistics needed to generate valid results, and perhaps look for differences between sub-populations (e.g. do managers and staff have the same or differing impressions about thud factor?  Do all departments and business units share more-or-less the same view?).

If we presume that a management-led team has been tasked with developing or reviewing Acme's security metrics, the PRAGMATIC approach would turn what is normally a rather vague and awkward argument over which metrics to use into a much more productive discussion about the merits of various candidate metrics, comparing their PRAGMATIC scores and using the individual ratings to propose improvements to their design.

More on survey metrics

Further to our last item about using security surveys judiciously as a source of metrics, today we're taking a gentle poke at the Data Breach Investigations Report (DBIR) published annually by Verizon Business.

DBIR is different to most published security surveys.  Arguably, it is not a survey at all.  It is based around the findings of information security incidents that Verizon have investigated.  To be more accurate and precise, the 2012 DBIR states (with our added emphasis):

"The underlying methodology used by Verizon remains relatively unchanged from previous years.  All results are based on first-hand evidence collected during paid external forensic investigations conducted by Verizon from 2004 to 2011.  The USSS, NHTCU, AFP, IRISS, and PCeU differed in precisely how they collected data contributed for this report, but they shared the same basic approach.  All leveraged VERIS as the common denominator but used varying mechanisms for data entry.  From the numerous investigations worked by these organizations in 2011, in alignment with the focus of the DBIR, the scope was narrowed to only those involving confirmed organizational data breaches."

If you read our previous piece about biased surveys, you can probably guess why we emphasized those terms:

• 'Underlying methodology' is a curious turn of phrase.  When is a method (or 'methodology', if you insist on having an ology!) not underlying?  Are they subtly trying to tell us something by pointing out that theirs was 'underlying' - like perhaps it wasn't in fact a pre-determined structured or formal method, but an 'approach' that evolved and was adopted informally and perhaps inconsistently as investigations took place?
• 'Relatively unchanged' presumably meant to imply that although there have been changes, they were inconsequential but what, precisley, has changed?  Shouldn't we be given the details in order to judge the implications for ourselves?  Otherwise, how do we know Verizon is not  attempting to gloss-over or ignore material differences simply in order to present long term trends without necessarily re-basing or discounting prior data?  
• 'Paid external forensic investigations' may be an important constraint on the report's applicability.  The population being sampled presumably just involves incidents that were deemed serious enough to warrant forensic investigation by a commercial specialist, specifically Verizon.  We would have to dig quite a bit deeper in order to determine how representative the sample is of all information security incidents and organizations, or simply accept that the findings may be valid only in that specific context.  
• That contributors to the report 'differed in precisely how they collected data' raises further concerns about the validity of the report.  Once more, we are not privy to the details.  In effect, we are being asked simply to trust that Verizon has carefully analyzed and combined the different data sets to eliminate any inherent bias and without introducing any further anomalies or statistical issues during the process.  From a strictly scientific perspective, that's quite an ask!  
• There is some ambiguity in respect of the source data.  First we are told that 'all results are based on first-hand evidence collected during paid external forensic investigations conducted by Verizon' but then the involvement of various other organizations is acknowledged: Verizon's part in investigating the incidents reported by those organizations is unclear.  If Verizon was directly involved in every case, as implied, why did they not do all the data entry and analysis themselves?
• 'Same basic approach' is almost meaningless: we guess they mean 'similar' but they aren't coming clean on the differences.
• VERIS deserves a deeper look - more on that later - but meanwhile we are left guessing about precisely what they mean by 'leveraged VERIS'.  Is 'leveraged' simply a fancy way of saying 'used', or is there more to it than that?  If the contributors used VERIS in substantially different ways, their information may not be directly comparable.
• 'Varying mechanisms for data entry' sounds quite trivial if it simply means the mechanics of data entry were different ... but perhaps not if there were more significant differences (for example, selective entry of certain information on specific cases someone deemed sufficiently "interesting", rather than entering data consistently across the board).
• Just how numerous were the 'numerous investigations' we wonder?  Is ten numerous?  A hundred?  Fifty thousand?  How many different organizations were involved?  What kinds of organizations were they?  How big/small?  What industries?  Which countries? ... These are pretty basic questions, the answers to which would materially affect the applicability of the reports metrics, analysis, findings and recommendations.
• In confirming that the 'scope was narrowed', we are reminded that the results may not be generally applicable, a concern we have already discussed.  
• Especially if taken out of context, the closing phrase 'only those involving confirmed organizational data breaches' would be highly misleading since in actuality the report apparently only concerns 'paid external forensic investigations' involving Verizon.  

Taking a step back, the paragraph raises serious concerns about the data sources and the analytic methods applied, and hence brings into question the validity and applicability of the results.

That's all very negative and cynical, of course.  You (and especially Verizon!) might feel we are being unfair in seeking to apply rigorous scientific principles to what is, let's be honest, a marketing product.  So, in the interests of redressing the balance a bit, we'll add that: 

• Verizon is a well-respected specialist in the field, a big player that can probably afford to employ good statistical experts/advisors.
• The DBIR has a record stretching back many years, hence the survey and analytical methods have hopefully been refined and improved over that period.
• Some of the issues we have raised are addressed, at least in part, elsewhere.
• Despite all the misgivings, the report is still useful.  It is well written, widely quoted, and available for free!  Make of it what you will.

We'll pick up on VERIS in another blog item but that's enough from us for today.  Meanwhile, your comments are welcome - over to you.

SMotW #32: asset management maturity

Security Metric of the Week #32: information asset management maturity

'Managing information assets' may not be the sexiest aspect of information security but it's one of those relatively straightforward bread-and-butter activities that, if done well, can substantially improve the organization's overall security status.  

The premise for this week's candidate security metric is that the management of information assets and the management of information security are related.  In mathematical terms, they are positively correlated.  A mature, comprehensive, well-thought-out and soundly implemented approach to the management of information assets is likely, we believe, to be associated with a high quality, effective approach to security management.  Even if the correlation is not terribly strong, asset management at least provides a solid foundation for assessing and ultimately managing the organization's information risks.

Conversely, weak or poor information asset management practices do not bode well for information security: if management has a rather loose grip on the organization's information assets, focusing myopically on its physical and financial assets while ignoring information in all its forms, it is unlikely to appreciate their value, understand the risks and be willing to invest adequately in the security needed to address them.

That's all very well, but how can information asset management practices be measured in practice by an ordinary organization such as Acme Enterprises Inc.?   One possibility is to adopt the maturity scale approach that we have discussed before in relation to measuring both business continuity and HR security practices.  Creating the maturity metric involves analyzing information asset management activities, picking out the key elements and then determining how they typically differ across the spectrum from bad to good practice.  Section 7 of ISO/IEC 27002:2005 ("Asset management") is a useful starting point, both in terms of the structure (laying out various responsibilities relating to information assets, including their identification, ownership, and classification) and the content (several security controls are recommended in this area). 

With a strong PRAGMATIC score of 86%, this metric is a strong candidate for inclusion in Acme's information security measurement system.

P	R	A	G	M	A	T	I	C	Score
90	95	70	80	90	85	90	85	90	86%

We have been a bit harsh on the Actionability criterion, figuring that it would be quite difficult in practice to persuade Acme's management to pay more attention to managing the organization's information assets, particularly if they were of the myopic persuasion noted above.  In certain circumstances, the metric alone may generate more heat than light on the security situation.  

Risky metrics from security surveys

Published information security surveys can be a useful source of metrics concerning threat levels and trends, although there are several different ones, each with different methods, sizes, scopes, purposes and designs.  Big, scientifically designed and professionally executed surveys are inevitably expensive, begging questions such as why anyone would fund them and publish the results, often for free.  What are their motives?  What's in it for them?

This is an important but seldom considered issue because of the distinct possibility of bias.  Bias is kryptonite to metrics.  Bias works like a catalyst: a small amount can have a disproportionately large effect on the outcome.

Some published survey reports are quite obviously biased, being little more than marketing vehicles that selectively collect, analyze and portray the information largely to promote their particular "solutions" as must-haves.  They tend to be based on rather small and non-random samples (which fact is never disclosed, except perhaps for someone reading-between-the-lines of the small print tucked out of sight) and the survey questions (which are not usually included in the reports, again for obvious reasons) are designed to encourage respondents to support certain presumptions or premises.  Even the way the metrics are presented tends to be skewed towards promoting a particular perspective - common examples being the use of compressed scale graphs (often with unlabeled axes!) and cut-pie-charts (over-emphasizing a certain segment by showing it cut and pulled out of the main pie).  These are patently not good metrics: there would be significant risks in basing our security risk/threat analyses and plans on such dubious data.  Nevertheless, interesting anecdotal information about certain incidents is often included in the reports, and arguably they have some value in general security awareness terms.  Despite the inherent bias, marketing copy and  blatant advertising has a useful role in making us think about the issues and consider the merits of various approaches to dealing with them - provided we are awake enough to realize that there is a strong agenda to promote and sell particular products.  

Other security surveys are more scientific although often it is quite difficult to determine how they were actually designed and conducted.  True, they often give a broad descriptive outline of the sample stating fairly obvious parameters such as the total sample size, but the sample selection methods, survey questions and analysis are generally not so easy to determine.  Some surveys do provide additional information in this area but most fall well short of rigorous academic/scientific standards, at least in their main published reports.  Nevertheless, the surveys that at least make some effort to describe their 'materials and methods' are more useful sources of security metrics, particularly the ones that are conducted by trustworthy industry bodies such as the Information Security Forum, or involve truly independent survey/statistics experts (no, I don't mean market research companies!).

Some surveys are produced and published as a genuine public service, generally by public bodies (who are presumably concerned that citizens are aware of, appreciate and fulfill their information security responsibilities, as well as wishing to gather and use the information for their own governmental strategic/security planning purposes) and/or public-spirited commercial companies (often consultancies with less to gain from promoting specific products aside from the general marketing/brand value in highlighting their own consultancy and auditing services to a global audience, of course).

Think about these issues the next time you are asked to participate in a survey.  Consider the manner in which you have been selected to participate, and if you go ahead with it, notice how the questions are phrased and presented.  Even something as innocuous as the sequence of multiple-choice answers can have a measurable statistical effect, while the choice of words in both the question preamble/stem and the answers can introduce a subtle (or not so subtle!) bias.  Watch out for key words, emotive phrases, and hints that the surveyor anticipates a certain response.   Recall your concerns when the results are finally published.

Finally, for today, when you are reading and thinking about using the metrics from published survey reports, think carefully about the credentials and motivations of the organizations behind them.  For example, check back through previous reports from the same outfit to see whether, in the light of experience, their trends analysis and predictions were broadly accurate and their recommendations were helpful.  Do they have a good record as surveyors?  How much can you trust them?

By the way, one of the clues as to the scientific validity and rigor of a periodic  survey concerns how changes to the survey across successive iterations are handled and reported.  Aside from checking whether prior data remain a valid basis for comparison and trends analysis if the wording of a question has materially changed, ponder the precise nature of the changes and how those changes are justified and explained.  If the survey's authors don't even take the trouble to discuss changes to the questions, ask yourself what they might be concealing.

Apollo metrics

If I had read about this a few years ago, I would probably have dismissed it as American propaganda, designed to fool the Russians into underestimating their technical capabilities:

"How each channel was generated, though, is almost shockingly primitive by today's standards. The computers downstairs in the RTCC were responsible for producing the actual data, which could be numbers, a series of plotted points, or a single projected moving point. The System/360 mainframes generated the requested data on a CRT screen using dedicated digital-to-television display generators; positioned over the CRT in turn was a video camera, watching the screen. For the oxygen status display example above, the mainframe would produce a series of numerical columns and print them on the CRT.

The numbers were just that, though. No column headings, no labels, no descriptive text, no formatting, no cell outlines, no nothing—bare, unadorned columns of numbers. In order to make them more understandable, an automated mechanical system would retrieve an actual physical slide containing printed column headings and other formatting reference information from a huge bank of such slides, and place the slide over a light source and project it through a series of lenses into the video camera positioned above the CRT. The mixed image, made up of the CRT's bare columns and the slide containing the formatting, was then transmitted to the controller's console screen as a single video stream."

That scarcely believable description in a fascinating article on ArsTechnica explains how vital real-time metrics concerning systems aboard or relating to the Apollo space missions were generated in the back rooms and displayed on the infamous consoles at "Mission Control, Houston".  

Suddenly, the quaint, amateurish almost home-made feel of the original bridge of the 23rd Century USS Enterprise appears rather more advanced, for its time, than perhaps we give it credit.

Rgds,Gary