Netflix's Cloud Contest: More Companies Should Follow Suit

The economics of open innovation are too compelling to ignore.

As is intended by such open contests, the winners weren't experts in movie recommendations, but in statistics, machine learning and computing. The Goldcorp Challenge turned around a Canadian gold mining company near bankruptcy to a recent market capitalization of more than $26 billion, and was won by a team of 3-D modelers and geoscientists with no prior familiarity with the geology of the mine. GE is running a challenge called Flight Quest in which it's releasing months of airline-related data in hopes someone can create an algorithm to augment pilots' decision-making processes in real time.

Some have taken issue with the original Netflix Prize, arguing that Netflix overpaid, the final prizewinning algorithm was never implemented, or that the business shifted from DVD rental to streaming. The algorithm that won the progress prize was tuned for Netflix's scale, and then wasput into production. The fact that the next set of incremental improvements was not implemented does not detract from the fact that the prize achieved its goal: a substantial improvement in the algorithm.

Moreover it is the nature of innovation that there are no guarantees of success. Getting additional informed perspectives on the limits of performance of recommendation algorithms is worthwhile: It's one thing for the lead recommendation algorithm designers at Netflix to say "this is the best we could do," and another to say "this is the best that anyone could do."

Finally, while a million dollars is not a trivial amount of money, it was a small investment compared to Netflix's $3.6 billion 2012 revenue, and equivalent to the fully loaded salaries of a handful of researchers for one year. That makes it extremely cost effective, even before accounting for the public relations value.

Often, ideas generated from innovation markets and contests such as this are used to enhance strategic advantage, acquire protected intellectual property such as patents and trade secrets, or enhance existing or create new products and services. This is fair: A company is paying for expertise and innovation in the form of a contest prize rather than R&D salaries. The learnings from the Netflix Prize could have been treated as proprietary and confidential, but instead were published; and to Netflix's credit, the same open sharing will apply to the Netflix Cloud Prize. After all, for the Netflix Prize, the value was not just in the algorithm, but the algorithm applied to Netflix's existing processes at Netflix's scale.

For the Netflix Cloud Prize, the value is not just in the tools being devised or enhanced, but applying those tools to improve the usability, quality, performance, reliability and security of cloud computing, a delivery model that permeates Netflix's operations. By leveraging an open source approach, Netflix is driving accelerated improvements to the cloud ecosystem, certainly assisting with its own service, but fostering a virtuous cycle of intercompany collaboration that benefits all.

The Web and the cloud have the power to support the growth of contests, both in creating virtual innovation markets that match challenges with contestants and in providing computing power on demand to support the cost-effective analytics that some contestants require to test their algorithms, whether recommendation engines, molecular dynamics, in-flight decisions or cloud software. I'm looking forward to November, when the prize winners will be announced at Amazon Web Services re:Invent.

At Cloud Connect Silicon Valley next week, Amazon and Netflix will be among the presenters discussing innovative ways to use the cloud. I'll be presenting, too. Hope to see you there.

Perhaps I missed it, but I don't see where Joe said the contest itself is a bad idea. He just said he found it overly AWS-centric. of course, that is, as you say Netflix's right. But Joe is a consistent proponent of open standards and portability (in fact, he just did long-form defenses of standardization and of cloud standards) and also has a right to push that agenda forward. The two are not mutually exclusive points. Lorna Garey, IW Reports

Enterprise cloud adoption has evolved to the point where hybrid public/private cloud designs and use of multiple providers is common. Who among us has mastered provisioning resources in different clouds; allocating the right resources to each application; assigning applications to the "best" cloud provider based on performance or reliability requirements.