Earlier today, this article from ComputerWorld came across my desk.  The headline grabbed my attention, having indicated controversy and disagreement, which of course I’m going to look into.  The article, which cites Microsoft’s semi-annual security intelligence report, claims that  Microsoft has only been right in it’s vulnerability exploitability predictions about 27% of the time.  Others quoted in the article purport that since their accuracy is so low, what’s the point?

They’re obviously missing the point, and I suggest that the premise of even trying to calculate such a metric as its accuracy is fundamentally flawed.

The numbers in the article, and really, any numbers you would care to calculate and be able to prove, can only be made using public information.  This means you count the number of exploits publicly known about, compare that to the number of vulnerabilities with a particular rating, and get your percentage.  This is what the article and the people it cites do.  This calculation, and it’s results, are useless.

If you read the “mission statement” from the top of the Exploitability Index page, you will find the following:

The Microsoft Exploitability Index is designed to provide additional information to help customers better prioritize the deployment of Microsoft security updates. This index provides customers with guidance on the likelihood of functioning exploit code being developed for vulnerabilities addressed by Microsoft security updates within the first thirty days of that update’s release.

Nowhere in that statement does it say anything about exploit use or disclosure, because it’s irrelevant to the point of the rating.  The rating is about how exploitable the vulnerability is, and whether or not exploit code is likely to be developed for it within thirty days, not whether or not it’s likely that such an exploit would get used, used widely, put in a product, partially or fully disclosed, posted to milw0rm, or anything else.  Granted, the next section on that page mentions the “release” of exploit code, but what does that actually mean?  It could mean any number of the list of actions that I just mentioned.  Microsoft couldn’t possibly hope to rate (guess, really) whether or not an exploit will surface publicly for a vulnerability and when.  All they have to make a determination with is the technical information about the vulnerability itself which is really only enough to make a determination about how difficult it would be to develop an exploit, not whether or not it will really happen or what the motivations of the person who does so will be toward disclosure after the fact.

Now in addition to all that, if you consider the fact that there are private exploits out there in the big bad scary world, any statistic you care to draw from the public exploit count is completely useless.  Because really, who cares all that much about the public exploits?  Sure they might get used more, but it’s the private ones that I’m far more worried about when considering defense.  Thinking back over the last few months of Microsoft Bulletins, I’m personally aware of a number of those vulnerabilities that have exploits written for them that are still not public, many of which likely never will be.  Granted, I have a more privileged view into this data pool than most people due to knowing some really smart and really talented people, but those people are only a small subset of those out there who are capable, and most of the people I know are not maliciously motivated.  If just my narrow view of what exploits exist shows this obvious difference in public versus private data sets, really, who knows how drastically different these two data sets really are?  How accurate the index is as a prediction is simply an impossible metric to even attempt to calculate.

People should stop trying to use the Exploitability Index as a mystic oracle that can predict the future, and use it for what it actually is, just another metric to consider when prioritizing patches.  Having a metric that indicates which vulnerabilities have a higher probability of having any exploit developed for them, public or private, is useful and is exactly what the index indicates.  It’s an informed classification of what could-be, nothing more.