How do you evaluate a quantum computer you just bought from D-Wave for $15 million? It's not easy, especially since no one can really understand how the machine -- with its ones, zeros and superpositioned "one-and-zeros" -- actually functions. Instead, all you can do is throw increasingly complex questions at it, and hope that it answers them quicker than a top-end classical computer. This quest for evidence of so-called "quantum speeedup" has been going on for a while, with little in the way of positive results. Now, a freshly-published collaborative study involving Google (owner of a D-Wave box), Microsoft (owner of some very advanced traditional tech), and a team of university scientists, has achieved new results that are equally disappointing. Science magazine describes the study as "the fairest comparison yet." D:Wave's founder, meanwhile, has described it as "total bullshit."

The dispute partly arises over the choice of questions that were run through each type of machine. The latest study deliberately used questions that both computers could readily answer, and D-Wave claims that these were too simple. The company has also pointed out that it's a "scrappy startup," whereas the rival classical machine had the benefit of decades of research and trillions of dollars of investment, so it's pretty remarkable that the quantum computer was even able to keep up. By contrast, the university-backed researchers say that their comparison was based on challenges that were tailored to be extremely easy for qubit-based processing, so there could and should have been evidence of speedup if quantum mechanics are truly being exploited.

It's not clear where things will go from here. The study has a conclusive-sounding tone, as does one of its lead researchers, Matthias Troyer from the Institute of Theoretical Physics in Zurich, who simply says "we don't see quantum speedup." Then again, Troyer acknowledges that other sorts of tests may reveal the phenomenon "eventually." Perhaps the onus is now on D-Wave to find some way to silence the skeptics, not just by keeping up with standard technology, but by surpassing it in a meaningful way. Whatever happens next, it sounds like relations between D-Wave and the scientific community are becoming fractured -- and that sort of tension will either speed progress up, or slow it right down.