Monday, March 07, 2005

The Researcher's Dilemma

I finally read Clayton's Christensen's The Innovator's Dilemma. A few years ago an NEC executive gave a talk using the ideas in the book to explain NEC's interest in quantum computing.

Christensen divides new technologies into two groups: sustaining and disruptive. Sustaining technology helps improve a product for its current customers for example Intel building a faster microprocessor. Disruptive technology is technology that is not initially useful for a big company's current clients. So other newer and smaller companies develop the technology for a niche market. But eventually the technology improves to meet the needs of the original big company's customers at a much lower cost. At this time the big company cannot catch up with the new technology and loses their main business to the newer startups. Christensen gives many examples such as the development of smaller disk drives and the advent of discount stores like Target hurting bigger retailers like Sears.

The book makes some solid arguments but determining which technologies will be disruptive is quite difficult. So companies need to place lots of bets perhaps why NEC funds quantum computing research just in case it becomes a disruptive technology in the future.

Do the same concepts apply to research? In complexity we have had some disruptive technologies, for example, the PCP theorem for hardness of approximation or the idea of Nisan and Wigderson of using hard languages to derandomize, or going way back the whole concept of NP-completeness. Other concepts like circuit complexity have not been as disruptive as originally thought. Also one could view tools like the probabilistic method or extractors as disruptive.

What could happen to large companies can also happen to researchers. Suppose George is a researcher who creates new results building on current ideas with small twists (sustaining technology). George ignores some new ideas in complexity because it doesn't seem to help him prove new results in his area. But as those new technologies develop they later allow others to go well beyond what George has done. Now George is behind the curve on the new disruptive technology and can no longer play an important role even in his own field.

What can George do? He can learn the new techniques or he can change fields. And often the newcomers never properly learn the old tools and George can still pull a few surprises out of his hat.


  1. George should stay abreast of the new field and continue extending the old field.

  2. Note that what you're describing is akin to Kohn's shift of paradigms. George may ignore it for a while, but eventually, once overwhelming results convince him otherwise, he'll already have tenure and be able to ignore it for the rest of his career... If he's a nice guy he'll try and learn some stuff and help the youngsters...

  3. Gilad, your "nice guy" sounds like my theory advisor. I wanted to work in "xxx" where "xxx" was considered purely a database issue at the time. He said "xxx is down the hall." So I did something that was an acceptable theory topic at the time for my Ph.D. Two years later, after more than a few articles had been published in theory journals on the topic of "xxx," by the leaders in the field, he had a student working exclusively on it....

    He is a nice guy. And not just for getting interested in "xxx," after the fact.

    I guess it's my fault for not being a better salesman.

    Word to the newbies who want to work in a new area: You have to sell your topic to your advisor. Keywords: sales, funding agencies, commerical interest, interesting theorems.