Doubt cast on Windows tests. But where was Microsoft?

So I wasn’t the only one scratching my head over claims made a week ago that real-world memory usage on Windows 7 could routinely hit the 90 percent mark. It turns out that the company from which the unusual claims emanated, Devil Mountain...

It turns out that the company from which the unusual claims emanated, Devil Mountain Software, has a CTO, Craig Barth, who turns out not to exist. Mr Barth, we now learn, is actually an outside blogger for IDG’s InfoWorld magazine (a sister title to Techworld) called Randall C. Kennedy.

Kennedy has since lost his job at InfoWorld, a few journalists at IDG have been made to look gullible, and a few others at ZDNet (the rival magazine stable that blew Kennedy’s cover after clever research) have been made to look whatever the opposite of gullible is. And Devil Mountain’s Windows data, some of which goes back years? The provenance of that is now in serious doubt.

I claim no deep insight into the workings of Windows and its memory design, but even to me the idea that a large base of pretty new Windows 7 PCs could average nearly 90 percent memory use struck me as far-fetched. A number of others were equally sceptical, mostly bloggers and expert users. To most journalists it was probably just another mildly contentious story on a busy day.

There’s quite a lot to pick apart here, and plenty of people to blame, but one element of the story that hasn’t been mentioned is the odd silence over the years from the company more often than not at the heart of the various Windows-related stories Devil Mountain has generated, namely Microsoft.

I can’t link to all the stories in question because they have now been pulled by most IDG titles, but in the ones I looked at Redmond was invariably silent on the claims being made. Even the most recent and hard-to-believe Windows 7 story garnered, as far as I am aware, not a single official sentence of response from the company.

Microsoft has a long-standing policy of keeping its press responses to a minimum, and even when it does say something it’s often gnomically unenlightening. The company sees itself as above getting drawn into meta-debates that would see it having to take a stance on a large number of contentious stories every week.

The trouble with this stance is it leaves journalists assessing Devil Mountain’s data at the foot of their own mountain, forced at best into raising questions on the basis of speculation, as I had to in my blog. That’s hardly ideal. Silence might be economical for Microsoft but it doesn’t serve its users well.

My own view is that the stream of eye-catching findings that have come out of Devil Mountain and Randall C. Kennedy are examples of the sort of promotional stuff many software vendors use to push their competence. It’s not that it’s nonsense, but it usually has a hidden theme that leads back to a product in some way.

In the case of Devil Mountain, this would be the idea that performance testing Windows and its apps tells us useful things that people and companies can act upon. Frankly, this runs counter to my own experience. PCs aren’t magic boxes of mystery. Getting one that runs Windows satisfactorily with a given set of tasks isn’t that hard.

The daftest aspect of this story is really that a tiny software outfit was able to generate so many performance-related stories about the most ubiquitous software in history, created by the largest software company in the world, because that company doesn’t publish any stats of its own. To my knowledge, Windows performance is talked of by Microsoft in very general terms. No data is ever produced to back anything up because Microsoft either doesn’t collect such data, or does collect it simply doesn’t put it into a form that outsiders would understand.

Readers of IDG’s stories quoting Devil Mountain’s Craig Barth didn’t know he was really blogger Randall C. Kennedy, but that was only one of a number of things they didn’t know. As it stands, we still don’t really know a lot about Windows performance either. It’s a matter of what we believe. I tend to see it as a side issue, but that’s just me. It’s the cloak of secrecy that serves nobody well.