Monday, November 14, 2005

Acceptance Rates versus Competitiveness

The acceptance rates at conferences for theoretical computer scientists tend to run higher than acceptance rates at conferences in other areas of computer science. Does this mean that theory conferences are less competitive than their counterparts in other areas? In a word, no.

Researchers look at their papers and for a given conference, either feel they have a very good chance of acceptance, a possibility of acceptance or a long shot of acceptance and tend to submit only if their paper falls in one of the first two categories.

In non-theory areas like artificial intelligence, the committee must take a subjective look at the papers which means many more papers fall into the second "possibility of acceptance" category. Many more people therefore take the risk and submit their paper because they can't immediately put the paper in that third "long-shot" category. This leads to more submissions and a low acceptance rate.

For theory we do a much better job putting our papers into these categories, as we can self-judge the hardness of the results and have a good feeling of the importance of the results for the conference. Theorists can tell when their papers won't have much of a chance of acceptance and will, usually, not waste their and the program committee's time in submitting these papers to the conference. This leads to a relatively higher acceptance rate in theory conferences.

A similar analysis would also explain why the funding rates for the theory NSF program also tends higher than the funding rates at many other programs in CISE.

20 comments:

  1. Was this prompted by something in particular? As a contrast, I think cryptography conferences tend to have a much lower acceptance rate than FOCS/STOC. Maybe we aren't doing a good enough job of pre-sorting submissions? or it could be that several distinct communities submit to the same conference sometimes.

    ReplyDelete
  2. In my area, software engineering, there are four kinds of papers:

    (1) good papers that get published; (2) bad papers that get published; (3) good papers that don't get published; and (4) bad papers that don't get published.

    It seems "a good" paper has a 50-50 chance of actually making it.

    ReplyDelete
  3. I think that one of the reasons why theory conferences have low acceptance rates, is, as you point out: self-selection. Authors judge for themselves if their paper is good or not for submission. I think that this is in large part because submission is not anonymous, and if people submit nonsense, the reviewers will know who submitted that nonsense.

    On on the other hand, conference such as SIGCOMM, SOSP, and several systems conferences have anonymous submissions. Thus, people can submit nonsense (which will never get accepted anyway), and get away with it. They do so thinking, "it won't get in, but I'll get some feedback from the committee anyway". That increases the number of submissions, and lowers acceptance rates.

    2006 sees CRYPTO going non-anonymous. I'll bet there is a fall in the total number of submissions this year :)

    ReplyDelete
  4. Lance -- I agree with you, or at least I would like to. You didn't point out that this argument is important to the future of theory (or at least of young theoreticians), since people now use conference acceptance ratios to judge work quality. (Listing acceptance ratios on CVs is becoming common, a practice I detest.)

    But really, isn't this argument painfully self-serving for our community? If we want to make the case that a FOCS/STOC paper with a 30% acceptance ratio is more significant than a conference in subfield XYZ with a 10% acceptance ratio, it seems like we need some evidence to back this argument up soundly. Otherwise, it just sounds like sour grapes.

    Michael

    ReplyDelete
  5. I'm always impressed with the high level of self-selection in theory conferences. In a recent large conference we had hundreds of submissions and less than 10% didn't belong there. All others had a reasonable shot at being accepted. In the end we had to select the top 30%, give or take a few, but save for that bottom 10% I would not have discouraged any of the other authors from submitting to the conference.

    ReplyDelete
  6. It would be nice to be able to point to concrete evidence of self-selection (which is clearly going on).

    Here's one possible measure: compare the acceptance rates of the generally acknowledged top conferences to those of the second tier conferences. The more similar they are, the more self-selection must be going on.

    ReplyDelete
  7. How can we gather evidence to test this self-selection theory?

    Perhaps the program committee from STOC or FOCS two years ago could go through the list of papers which were then rejected, and find out what happened to them -- whether and where they eventually got published.

    I would hope to see something along the following lines:
    -submitted 250
    -accepted 75
    -accepted at a later STOC/FOCS 5
    -accepted at Soda, Crypto, Complexity, Socg, or Podc (or some other similar "best conference in its subarea of theory" conference that I am forgetting) 100
    -accepted at a "specialized workshop respected by people in that area" 40
    -unpublished 25

    To me, this would be evidence both of the accuracy of the decisions reached (only 5 false negatives) and of the self-selection process (most papers submitted are strong enough to be published in the best conference of their subarea).

    ReplyDelete
  8. Claire's idea sounds good, although I would debate the 'false negative' interpretation. The feedback from one conference can lead to a much better version submitted to a subsequent conference.

    ReplyDelete
  9. That is true. On the other hand, after first STOC/FOCS rejection, some people might decide not to submit their paper to those conferences again, so the "false negative" rate could be actually higher.

    ReplyDelete
  10. One of the many possible reasons may be as follows:

    For a theory paper, every co-author get the credit (at least in theory). However, in almost every other CS subfields, only the first author get the majority of the credit (a student-advisor collabaration is an exception)

    Therefore, for the folks to survive in non-theory CS subfields, they have to write more papers, to submit a paper again and again till accepted, ...

    ReplyDelete
  11. Did this topic come up because
    some one suffered at tenure or
    promotion time because of the high
    acceptance rates of theory
    conferences?

    ReplyDelete
  12. How do you know self-selection is not practiced in other conferences as well?
    How familiar are you, Lance, with the history of submissions and acceptences in top systems conferences like SOSP and SIGCOMM?
    Have you seen the whole roster of submissions to any of these (or similar) conferences?

    Did I see the phrase "self-serving" before?

    The theory community should carefully do a reality check for itself: there is no reason to assume there are many so-so papers floating around and not submitted.
    The real issue is that STOC has become too narrowly-centered on compelxity, driving algorithms to SODA and other topics to other conferences (which get lots and lots of submissions).

    ReplyDelete
  13. This last sentiment seems false to me. I do not know any algorithms researcher who, thinking his or her paper is of very high quality, would choose to submit to SODA _instead_ of FOCS or STOC.

    I do know many senior (i.e. tenured) people who will submit a paper to SODA just because of the timing of the submission deadline instead of waiting for the next STOC. But the main point is that there is no conflict; no one has to decide between STOC/FOCS/SODA since the decisions for FOCS always come out before the SODA deadline.

    The Complexity deadline, on the other hand, is usually in the neighborhood of the STOC deadline, so it makes sense that complexity researchers have to make tough decisions every November.

    ReplyDelete
  14. I know many instances of (non-tenured) people submitting high quality papers to SODA rather than STOC/FOCS simply because they solve a problem in the three months between April and July. If they think someone else is working on the problem, too, then obviously they will submit to SODA. The previous poster does not have a clue.

    ReplyDelete
  15. The anonymous from comment 12.1:06 AM, November 17, 2005

    In short, for several other areas, people rank the "specialized conference" at the same level as STOC/FOCS. Hence many (junior and senior) people don't wait for STOC/FOCS.

    In certain cases, people prefer to publish in the spcialized conferences as they provide better exposure to the relevant community.
    So with certain types of papers, some people even wait for the specialized conference even if they think the paper is of high quality.

    There's no reason anymore to treat SODA/PODC/SPAA/CG to be lower on the "food chain" than STOC/FOCS.

    ReplyDelete
  16. This is ridiculous that someone compares SODA with Complexity (SODA
    has more than 450 submissions, I do not think that Complexity has
    even 1/4 of this). Nowadays lots of people do not wait for
    STOC if they have the results ready for SODA, esp. since STOC is
    narrowly-centered on complexity. I have seen lots of papers in SODA
    appearing in JACM or SICOMP which are surely more prestigious than
    FOCS/STOC. I really don't think that SODA level is too different from STOC/FOCS level at least nowadays.

    ReplyDelete
  17. As an algorithms researcher, I prefer getting a paper in SODA over STOC/FOCS. STOC/FOCS is definitely more complexity oriented and doesn't draw near as many algorithmic researchers as SODA.
    Thus an algorithmic paper in SODA has a wider impact than one in FOCS/STOC.

    ReplyDelete
  18. Soda vs. Stoc/Focs has been discussed on this blog at length just recently, and we don't need to take up that discussion again.

    We have no hard data and anonymous opinions are of tenuous value at best. Is everyone aware that they can post a non-anonymous comment by choosing from the "Choose an indentity" menu when they leave their comment?? Lance, perhaps you should give us your thoughts about anonymous vs. non-anonymous contributions...

    ReplyDelete
  19. Clair, I think the acceptance ratio is also quite related to SODA vs. FOCS/STOC issue, since the number of submissions in SODA is relatively high and in the same time SODA is one of the best theory conference.

    ReplyDelete
  20. Recently, a person who had claimed during the previous discussion that STOC/FOCS does not look down on algorithmic papers said during a conversation "algorithms is not theory".

    Sadly, I don't think he's an isolated point. He represents a small but non-negligible percentage of the theory community who either thinks that algorithms is not theory or algorithms is "theory gone wrong".

    I believe such views are on the way out, and have been on the wane since the mid 90s. However before that they were the dominant view of the STOC/FOCS program committees and even to date still some quite prominent members of STOC/FOCS PCs believe that SODA won't be an equal to STOC/FOCS until it has an equal percentage of fiendishly complicated complexity-style results. I think the term they use to describe them is "elegant", while others would describe them as "overwrought with technical challenges" instead.

    ReplyDelete