Thursday, June 03, 2010

Conference Acceptance Rates

In the latest CACM, Jilin Chen and Joseph Konstan analyze data from the ACM DL and conclude that low-acceptance rate conference have papers with higher impact (more citations). This shouldn't be too surprising. If Joe believes a conference A bestows more prestige on their paper than conference B, Joe might submit his paper to A instead of B if it Joe's paper had a higher probability of being accepted into conference B. So in equilibrium conferences with higher prestige papers should have a lower acceptance rate.

Chen and Konstan suggest that conferences could lower than acceptance rate to get higher prestige. I have two problems with that advice. First of course my general annoyance of treating CS conferences as "journals that happen be be held at a hotel". 

Chen and Konstan have the causation effect backwards. Prestigious conferences do have a lower acceptance rate with other factors kept the same. But you can't necessarily up the prestige by changing the acceptance rate. The one decision the PC can make is how many papers get accepted into the conference. But accept fewer papers and in the future less people will submit their papers since they no longer believe they have a decent chance of seeing their papers accepted. it's possible in the long run to increase the acceptance rate while lowering the number of papers accepted.

When STOC moved to double sessions and accepted more papers, did the acceptance rate dramatically go up? No, because we had more submissions to match. But why does STOC have a higher acceptance rate than similarly rated conferences in other subfields? Theoretical work by its nature is easier to self-judge and most people whose papers are well below the accept/reject border of quality don't bother to submit to STOC.

You have a prestigious conference by the people that come. You can't engineer prestige, it has to be nourished.

9 comments:

  1. Impact factor is citations/paper count, so if the acceptance rate is lower, the pool has been cut to not include papers that may lower the average citations per paper.

    This seems more like a direct law of statistics than sociology, but your points remain valid.

    ReplyDelete
  2. I think a large part of it is who is on the PC. If you read the list and you realize you've only vaguely heard of two of the people, you know it's not very prestigious. On the other hand, if you recognize many of the most important people in your field, you know that your submission has the potential for impact, or that at least you might get some worthwhile feedback.

    I think an author would think about that before thinking about the acceptance rates.

    ReplyDelete
  3. The problems would come if every conference did this; it would just lead to the unnecessary creation of more conferences and more needless busy work for everyone.

    ReplyDelete
  4. I think that one could also legitimately question the use of citation count as a measure of a paper's impact. While having a high number of citations is probably a necessary condition for a paper to be deemed significant, it is not so obvious that it is a sufficient condition.

    ReplyDelete
  5. In CS theory unlike math and many other areas of CS there is a confounding aspect of impact factors in that work is usually published in two citable forms, both the conference and journal versions.

    ReplyDelete
  6. Math Researcher3:36 PM, June 03, 2010

    Impact factor = citations/paper in the past 2 years. Therefore it should more aptly be called the "immediate impact factor". It should not be confused with how good, important, or even interesting the results are in a paper, because no one has the insight/foresight into how each piece fits into where in the grand scheme of the research landscape. I frequently cite math papers from decades ago, for example.

    Like the mathematician Steven Krantz says: "There are prestigious journals and there are excellent journals." Perhaps the same can be said about TCS conferences?

    ReplyDelete
  7. "I think a large part of it is who is on the PC. If you read the list and you realize you've only vaguely heard of two of the people, you know it's not very prestigious."

    yeah, like the PC member will *not* ask their phd students or posdocs to review the papers....

    it doesn't matter the PC member list, they are just flashing their names in the conference webpage in the end of the day...

    good luck sending papers to conferences based on PC members listings, that is a surefire way to detect its prestigiousness

    ReplyDelete
  8. where are the attacks on lance's writing errors?

    ReplyDelete
  9. To the previous commenter:

    People just like to have fun with GASARCH.

    ReplyDelete