So he collected more than 200,000 “Jeopardy” questions, took a random sample of 10,000 of them, and fed them verbatim — without quotes — into a few search engines, including Google and Microsoft’s Bing. To count, the answer had to appear in the first page of search results.

Both search engines performed surprisingly well — and better than the average human.

On Google, Wolfram found, the correct answer came up 69 percent of the time. Ask.com ranked next with 68 percent and Bing recorded 63 percent.

Bing edged even closer when he counted only the first search result. Google was dead-on 66 percent of the time, and Bing got 65 percent of the questions right.

Wolfram|Alpha chart

Wolfram wrote about the experiment in a blog post about the overall Watson project. Here’s an excerpt:

If nothing else, this gives us pretty interesting information about the modern search engine landscape. In particular, it shows us that the more mature search systems are getting to be remarkably similar in their raw performance — so that other aspects of user experience (like Wolfram|Alpha integration) are likely to become progressively more important.

But in terms of Jeopardy, what we see is that just using a plain old search engine gets you surprisingly far. Of course, the approach here isn’t really solving the complete Jeopardy problem: it’s only giving pages on which the answer should appear, not giving specific actual answers. One can try various simple strategies for going further. Like getting the answer from the title of the first hit — which with the top search engines actually does succeed about 20% of the time.