Wednesday, March 18, 2009

The Best or the Worst

Do you judge a conference by its best or its worst? I don't mean the absolute best paper, any conference can get lucky. I don't mean the absolute worst, every PC makes mistakes. But do you judge a conference more by the top quarter of its submissions or its bottom quarter?

Ideally you should care about the best papers. You only have time and energy to go to a fraction of the talks anyway so you can just skip the weaker papers. You'll learn the most from the best papers in the conference.

But how do you distinguish the best from the worst? The program committees purposely don't releases any ranking information of the accepted papers beyond a few award winners. So if the worst papers are pretty good this gives a lower bound on any talk you attend. 

But the real answer lies in the fact that we don't care about conferences because of the papers that we want to see but for what it does for our papers if they appear there. If the worst papers are very good and your paper gets accepted this implies some quality level about your paper. And in the end we want conferences that make our papers look good. And so we focus more on the worst than the best. Yet another paradox of the CS conference system.

8 comments:

  1. Neither. It is "How good is the typical paper?" which also says something about "Will I find the good people I want to talk to?".

    ReplyDelete
  2. How is this a paradox?

    ReplyDelete
  3. Do you judge a conference by its best or its worst?

    Why is this adolescent urge to "judge" a conference ? The measure of success of a conference is how many and what sorts of collaborations it produced -- might be even between people from different areas who wouldn't have conversed otherwise. This emphasis on "judging" is rendering theoretical CS conference into beauty contests and
    shifting the emphasis away from the actual scientific activities that should occur in these conferences.

    Here's one suggestion. Get rid of the proceedings (as well as the "best this and best that" awards that go with them). Or at least make sure that the proceedings have no archival value by limiting distribution solely to the attendees. Apart from reducing costs this would shift attention somewhat away from the beauty contest aspect. Those people who feel that their work is worth archiving should submit their papers to journals -- as is the norm in serious scientific disciplines.

    ReplyDelete
  4. An important function of conferences in CS (and journals in other areas) is exactly to provide the service of filtering a few papers out of the many written. Otherwise, how would we all know which papers to spend our limited time reading? Without any mechanism for filtering most of us would never get a signal that we should spend some time reading a paper by an unknown student. As dissemination is not an important service of conferences anymore since everything is on the web, the filtering service gains importance.
    In this vein, Oded Goldreich offers a personal "paper Filter" to serve as an alternative. Blogs can be a nice alternative as well

    ReplyDelete
  5. An important function of conferences in CS (and journals in other areas) is exactly to provide the service of filtering a few papers out of the many written.

    Then why have a physical conference at all , if filtering is the objective. Also, I am not quite sure that the learned journals in various fields consider "filtering" as their service to their respective communities. I suspect that they think of "archiving the most important results and contribution for easier use later on" as their main reason for existing.

    Also, many seminal results in mathematics (for example) never make it to journals at all -- they are published in proceedings of one-off conferences, monographs, or not published at all (!) ("Konsevitch's lecture at Orsay, 1995" is now a standard reference in motivic integration theory). Clearly, if one is "filtering" and skimming only articles in top journals, one would not notice most of the major developments in mathematics.

    ReplyDelete
  6. Anon #5 has it right. If filtering is the objective then we do not need a physical meeting. All we need is to clone John Baez's "This week in mathematical physics" into "This week in TCS".

    The original post by Lance also overestimates the quality of the bottom 30% of papers. Open the proceedings of a late 80's or early 90's FOCS/STOC proceedings and you'll find plenty of irrelevant "me too" and "minor variation on topic du jour" papers that would not make it past the first round of the PC meeting today.

    ReplyDelete
  7. Also, many seminal results in mathematics (for example) never make it to journals at all -- they are published in proceedings of one-off conferences, monographs, or not published at all (!) ("Konsevitch's lecture at Orsay, 1995" is now a standard reference in motivic integration theory).


    This occasionally happens, but "many" is an exaggeration. Kontsevich is a famous example precisely because it's quite rare. When it does happen, it typically involves very famous people who don't feel the need for any external validation or vetting for their work. If you are widely considered one of the top few dozen mathematicians in the world, you can easily get away with this. If you are merely an excellent mathematician (likely to get tenure at a top department but never in serious competition for a Fields medal) you probably can't.

    ReplyDelete
  8. This occasionally happens, but "many" is an exaggeration.

    The original intent was to point out that at least in mathematics the Journals are not complete repository of current research -- and if one just skims journals for this purpose one will probably miss much of the action. The fact that only famous people can get away not publishing in journals does not contradict this.

    I don't think that Konsevitch is an isolated case. For example, Grothendieck's original papers (such as the famous "Tohoku paper") or for that matter the SGAs and the EGAs -- basically the whole foundations of modern algebraic geometry -- were never published in mainstream journals.
    For that matter did Laurent Schwartz
    publish any journal papers on distribution theory aside from his book ?
    Hormander got his Fields medal for work reported in his book -- Linear Partial Differential Operators. I don't think de Branges published his proof of the Bieberbach Conj. either in a journal.
    More recently, Perelman did not publish his proof in a journal.

    More I think of it, I think contrary to what people think, the fundamental math breakthroughs are (more often than not) not published in journals at all.
    Is that right ?

    ReplyDelete