Computational Complexity and other fun stuff in math and computer science from Lance Fortnow and Bill Gasarch

Monday, March 07, 2005

The Researcher's Dilemma

I finally read Clayton's Christensen's The
Innovator's Dilemma. A few years ago an NEC executive gave a talk
using the ideas in the book to explain NEC's interest in quantum
computing.

Christensen divides new technologies into two groups: sustaining and
disruptive. Sustaining technology helps improve a product for its
current customers for example Intel building a faster
microprocessor. Disruptive technology is technology that is not
initially useful for a big company's current clients. So other newer
and smaller companies develop the technology for a niche market. But
eventually the technology improves to meet the needs of the original
big company's customers at a much lower cost. At this time the big
company cannot catch up with the new technology and loses their main
business to the newer startups. Christensen gives many examples such
as the development of smaller disk drives and the advent of discount
stores like Target hurting bigger retailers like Sears.

The book makes some solid arguments but determining which technologies
will be disruptive is quite difficult. So companies need to place lots
of bets perhaps why NEC funds quantum computing research just in case
it becomes a disruptive technology in the future.

Do the same concepts apply to research? In complexity we have had some
disruptive technologies, for example, the PCP theorem for hardness of
approximation or the idea of Nisan and Wigderson of using hard
languages to derandomize, or going way back the whole concept of
NP-completeness. Other concepts like circuit complexity have not been
as disruptive as originally thought. Also one could view tools like
the probabilistic method or extractors as disruptive.

What could happen to large companies can also happen to
researchers. Suppose George is a researcher who creates new results
building on current ideas with small twists (sustaining
technology). George ignores some new ideas in complexity because it
doesn't seem to help him prove new results in his area. But as those
new technologies develop they later allow others to go well beyond
what George has done. Now George is behind the curve on the new
disruptive technology and can no longer play an important role even in
his own field.

What can George do? He can learn the new techniques or he
can change fields. And often the newcomers never properly learn the
old tools and George can still pull a few surprises out of his hat.

3 comments:

Note that what you're describing is akin to Kohn's shift of paradigms. George may ignore it for a while, but eventually, once overwhelming results convince him otherwise, he'll already have tenure and be able to ignore it for the rest of his career... If he's a nice guy he'll try and learn some stuff and help the youngsters...

Gilad, your "nice guy" sounds like my theory advisor. I wanted to work in "xxx" where "xxx" was considered purely a database issue at the time. He said "xxx is down the hall." So I did something that was an acceptable theory topic at the time for my Ph.D. Two years later, after more than a few articles had been published in theory journals on the topic of "xxx," by the leaders in the field, he had a student working exclusively on it....

He is a nice guy. And not just for getting interested in "xxx," after the fact.

I guess it's my fault for not being a better salesman.

Word to the newbies who want to work in a new area: You have to sell your topic to your advisor. Keywords: sales, funding agencies, commerical interest, interesting theorems.