You would think that an organization committed to reporting lists of software-vendor security vulnerabilities—unpatched critical and high-risk bugs—would be unfailingly rigorous in its procedures and policies when compiling and distributing that information.
You would think that before crowning itself with a name like the “X-Force Team,” such an organization would be brutally meticulous in avoiding incomplete information, imprecise methods, and anything else that could lead to inaccurate results.
You would think that a global IT powerhouse with IBM’s reputation, resources, and credibility would never even think about getting into such a high-profile and high-stakes undertaking unless it the entire process, from end to end, was completely transparent and free of doubt, innuendo, and misinterpretation.
And you would be wrong, wrong, and wrong.
As it turns out, IBM’s X-Force Team might well have carried out its work with nothing but the very best of intentions, but its processes in putting together its lists of software vendors with the highest number of vulnerabilities was itself plagued with flaws and shortcomings—what we might indeed call vulnerabilities.
All of this came to light this week after the IBM X-Force Team’s mid-year list of highest-vulnerability software companies came out with Google at the very top of the list with an ugly showing of 33% of its critical and high-risk vulnerabilities unpatched. Any CIO looking at that would have to wonder what the heck was going on at Google, and would have to at least consider reevaluating the wisdom of expanding commitments to Google software.
But as my colleague Kelly Jackson Higgins of our sister site Dark Reading points out in her news analysis of the mid-year list, it turns out that the X-Force Team’s findings on Google were wildly off the mark and that instead of having failed to patch fully one-third of its vulnerabilities, Google in fact had absolutely zero such critical and high-risk bugs to fix in the first place!
But unfortunately the story gets worse:
What I found to be even more troubling than that embarrassing flub—to err, after all, is human—were the fairly casual comments after the fact from IBM’s X-Force Team saying in effect that, well, yeah, our data-gathering methods weren’t exactly tip-top, and sure, we realize that a lack of standards made these efforts and our results a little fuzzy.
In fact, the X-Force has now decided to X-punge just about every single one of its processes, according to a blog post from X-Force Research manager Tom Cross that Jackson Higgins cited in her article:
“As a consequence of this feedback, we have manually reassessed the CVSS scoring, remedy information, and vendor information for every vulnerability that impacted the percentages that appear in this chart,” Cross said in his post.
As they say, other than that, the chart was just fine.
I don’t know the X-Force Team but, again, I’m perfectly willing to believe their intentions were great and that at times—when their information was not totally off the mark—their reports could have been of some help to some companies. But a look at some of the comments from the Google security expert who worked with IBM’s Cross to rectify the inaccurate findings on Google reveals a set of processes that were terribly flawed.
From Jackson Higgins’ article, here’s Google security team member Adam Mein on both the specific inaccuracies relating to how Google was represented in the mid-year report, and on the way the report’s findings are put together.
“We questioned a number of surprising findings concerning Google’s vulnerability rate and response record, and after discussions with IBM, we discovered a number of errors that had important implications for the report’s conclusions. IBM worked together with us and promptly issued a correction to address the inaccuracies,” Mein says in a blog post excerpted by Jackson Higgins for her article.
Google’s Mein goes on to say that “We learned after investigating that the 33% figure referred to a single unpatched vulnerability out of a total of three — and importantly, the one item that was considered unpatched was only mistakenly considered a security vulnerability due to a terminology mix-up. As a result, the true unpatched rate for these high-risk bugs is 0 out of 2, or 0%.”
Mein later recommended a number of steps that should be followed by X-Force or any other outfit engaged in tracking and reporting vulnerabilities, according to Jackson Higgins’ article:
“As a first step, database compilers should reach out to vendors they plan to cover in order to devise a sustainable solution for both parties that will allow for a more consistent flow of information. Another big improvement would be increased transparency on the part of the compilers — for example, the inclusion of more hard data, the methodology behind the data gathering, and caveat language acknowledging the limitations of the presented data.”
Those ideas make even more sense when you consider some of the other corrections to the mid-year report that the IBM X-Force Team had to make. From Jackson Higgins’ article:
• “And Sun went from 24 percent of unpatched bugs of all severity to 8 percent, and from 9 percent unpatched critical and high-risk ones to zero percent.”
• “Microsoft also fared better in the corrected data, with 7 percent unpatched critical and high-risk vulnerabilities versus 11 percent in the older version. Its percentage of bugs of all levels of severity stayed the same at 23 percent.”
But perhaps there’s some cosmic justice in all of this, because after all the revisions and corrections, it turns out that the company holding the uncoveted #1 spot on the list of most unpatched critical and high-risk bugs is none other than IBM, with 29%.
Good intentions are great, but they don’t make up for the harmful repercussions from bad information, incomplete processes, faulty assumptions, and a lack of rigor, particularly in the context of such sensitive and vital findings. It might be a good idea for IBM and its X-Force Team to scrap even the consideration of issuing another report until they’ve tightened up their processes and results to the point where they’ve won the unwavering confidence of the software vendors they’re analyzing and the enterprise customers who base critical decisions on those findings.
The headline above this column uses the word “ignorance” and by that I mean the literal definition of the term: “lack of knowledge, learning, information, etc.”. The X-Force Team is no doubt stuffed with brilliant people, but when they’re dealing with incomplete information and are aware of uncertainties inherent in what they’re reporting—as was clearly the case with this mid-year report—then it’s clear they’re operating from a position of a “lack of knowledge, learning, information, etc.”—to wit, ignorance.
The IT marketplace—software vendors, enterprise customers, and security specialists who work rigorously to make the online world a safer place—deserve a lot better than that.
Source:http://www.informationweek.com/news/global-cio/interviews/showArticle.jhtml?articleID=227300114&cid=RSSfeed_IWK_News
