Computational Complexity and other fun stuff in math and computer science from Lance Fortnow and Bill Gasarch

Thursday, May 03, 2012

Microsoft saves the Yahoo NY Researchers

I started working with David Pennock on prediction markets back when we both were at the NEC Research Institute in New Jersey a decade ago. After a major reorganization the dropped basic research from their mission, I went back to academics but David stayed in industry research first at Overture which soon was swallowed up by Yahoo. He ended up at Yahoo Research New York in a small but amazingly strong research lab including machine learning theorist John Langford and social scientist Duncan Watts. But with a new Yahoo CEO and Prabhakar Raghavan and Andrei Broder's departures for Google, it became clear that the days of Yahoo research were numbered.

Today Microsoft announced that they are hiring 13 researchers from the Yahoo lab including David, John and Duncan as they start a new Microsoft Research Lab in New York, initially led by Microsoft New England director Jennifer Chayes. Lots of nice coverage from the New York Times, All Things D, blog posts from Jennifer and John, and a pictorial take from Chris Maase. Not the first time Microsoft has done something like this, Microsoft Research Silicon Valley got its start by hiring several researchers from the old Xerox PARC.

I'm happy the Yahoo researchers found a great home but I also mourn the loss of yet another company abandoning basic research in computer science.

12 comments:

It is sad, but most companies don't see the need any more. Google, Apple, Facebook, none of them do any basic research, although Google at least does some applied work. Yet we keep on churning out PhDs with no job prospects for doing research.

Foreseeably, at least some academic institutions will embrace tick-tock practices … and at these institutions, traditional academic values and practices will change substantially.

Like Intel relative to other STEM corporations, tick-tock academic institutions will seek a competitive advantage over traditional academic institutions by evolving more rapidly, by tapping new revenue streams, by creating more jobs in their local communities, and by attracting more ambitious students.

Whether tick-tock acceleration will be good or bad for STEM academia overall, remains to be seen … but definitely some institutions will attempt it.

I don't think it is "research" per se that companies are dropping, but rather the idea that research should happen in a research unit separate from the rest.

Which basically amounts to turning research into a glorified form of development. Allowing people with PhDs to call themselves researchers is a useful recruiting tool, but what's important is what those researchers actually do. In most companies, the answer seems to be advanced development that goes beyond the undergraduate curriculum. This is a fine thing to do, but it's not research as understood in academia.

Anonymous, your arguments echo the arguments advanced against the Human Genome Project, which similarly (some folks argued) was "a glorified form of development" that was not "research as understood in academia."

For better or for worse, it is inarguable (IMHO) that "research as understood in academia" does not scale to a planetary population of $7\times 10^9$ people. For example, if $1/10,000$ citizens is a CS researcher, each attending two conferences per year, and publishing four research articles per year, then how many CS conferences and CS research articles is that … per day?

For better or for worse, it is inarguable (IMHO) that "research as understood in academia" does not scale to a planetary population of $7\times 10^9$ people.

I'm about to disprove the inarguability by arguing the point. In fact, I'll go so far as to say that academic research will never face any serious scaling limits at all. Of course you're right that conferences work best for a limited number of people, not too geographically dispersed, but this's hardly a fundamental limitation on research.

Unless we experience huge population growth or a massive increase in the fraction of people involved in research careers, we can expect an order of magnitude increase in the size of the research community (probably over a 50+ year timeframe, as developing countries build research strength), and then that's about it. That's a lot of growth, but it's small compared to the growth so far in history. The community will expand to cover a broader range of topics, and there will be more specialization, but increasing size will not threaten the viability of academic research.

Counting papers per day is irrelevant, since there are already far too many for anyone to read. History is full of people saying "Oh no, right now I can barely read all the important papers, and in the future it will clearly be impossible." That is completely right but utterly unimportant. Life will go on, and we'll adapt just fine, like we always have. It's too bad we can't read all the papers in all the scientific fields, but those days are long gone.

Anonymous, please let me say that I appreciate and respect both the substance and style of your well-reasoned & well-expressed post.

And yet, it may happen that new academic research norms will emerge in coming decades, and as authority for this view, I will quote from Stephen B. Johnson's The Secret of Apollo: Systems Management in American and European Space Programs (2002):-------------"Believing that humans are irrational, I find the creation of huge, orderly, rational technologies almost miraculous. I had never pondered the deeper implications of cooperative efforts amid irrationality and conflict, and this project has enabled me to do so."

… "I sincerely hope that this work helps others recognize that the "systems" in which we all take part are our own creations. They help or hinder us, depending upon our individual and collective goals. Regardless of our feelings about them, they are among the pervasive bonds that hold our society together."

… "In a hotly contested Cold War race for technical superiority, the extreme environment of space exacted its toll in numerous failures of extremely expensive systems. Those funding the race demanded results. In response, development organizations created what few expected and what even fewer wanted -- a bureaucracy for innovation."-------------

Perhaps 21st century CS may evolve so as provide foundations what the 20th century space program foreshadowed: not so much a "bureaucracy for innovation" as a (much nicer sounding!) community for enterprise.

Thanks Lance. It's sad to leave behind Yahoo and an extremely enjoyable and rewarding part of my career but I am beyond thrilled at the outcome. Just a clarification: it's not clear whether Yahoo is totally "abandoning basic research in computer science". Time will tell. Yahoo Labs will likely live on in some form, probably more applied but perhaps not solely so. Ron is fantastic and may be able to do something special there. The Barcelona group remains largely intact and just got 7 (!) papers into SIGIR. Some other groups remain as well.

"Not the first time Microsoft has done something like this, Microsoft Research Silicon Valley got its start by hiring several researchers from the old Xerox PARC." Actually, the nucleus of that lab came from the Compaq (formerly DEC) Systems Research Center.