From the latest CACM, a short article describing a wiki set up by the ACM Health of Conferences Committee to discuss best practices for running conferences. As I write this the wiki is empty, a victim of spam attacks.
The article mentions several selected ideas for computer science conferences in general with some of my TCS oriented viewpoints.
- Accepting More Papers. What is the right acceptance rate for a conference? The article suggests 20-30%, about where we have it for many theory conferences. We don't focus so much on acceptance rates, it tends to happen automatically. If we add more talk slots, then more people tend to submit.
- Visionary Venues. Showcasing papers that present more farsighted or creative ideas. Some theory conference have informal rump and open problem sessions. Should we do more?
- Author Responses (Rebuttals). Allow the authors to provide the program committee responses to reviewers concerns. In my opinion this will add too much to the reviewing time and if the authors cannot properly express their ideas, they can update their paper for the next conference.
- Competitions. For example the Electronic Commerce conference runs competitions on various automated trading strategies. Not particularly applicable to theory conferences.
- Tracking Reviews. Passing reviews on a paper from one conference to another. Sounds too complicated for us since we have so many different overlapping conferences.
- Two-Phase Reviewing. Some conferences have introduced a two-phase review process where papers with an obvious bug, obviously non-novel or out of scope are rejected with a less rigorous review than those that are competitive. This happens already in theory conferences as it is much easier for us to separate the wheat from the chaff.
- Double-Blind Submissions. Keeping authors and/or reviewers anonymous.
- Hierarchical Program Committee. Theory conferences still aren't large enough to warrant this structure.
- Co-Located Workshops. Helps boost attendance with more focused programs. We should also have more joint conferences, even just two in the same location as well as the occasional monstrous FCRC.
I am feeling opinionated today so I will comment on all of the above.
ReplyDeleteAccepting More Papers: Should never be a criterion in itself, but it is good to see that our community is self-regulating.
Visionary Venues: I would like more rump sessions in conference. As for "showcasing papers...", no, the only criterion for submitted papers should be excellence.
Author Responses (Rebuttals): No. I second the idea that conferences should be 1-shot as there is always the next conference. Some committees approach authors in special cases where they feel that they need more info to make a decision, which is Ok and good enough.
Competitions: We already have one, it's called "getting your paper into the special issue". I also like the practice of Best Paper and Best Student Paper.
Tracking Reviews: Certainly not. An author should be able to get a review that is unbiased by a previous one.
Two-Phase Reviewing: I believe that in some sense this is already done informally.
Double-Blind Submissions: No. Actually I do want to use knowledge on the authors to help judge in some cases. Besides, it is too easy to circumvent a double-blind process from both sides.
Hierarchical Program Committee: I second the opinion that our conferences are not large enough, and I hope it will stay this way. If the community grows, then the conferences should grow in number rather than size.
Co-Located Workshops: For it to work it is better done outside the semesters where there are less time-limits for the attendants. Personally I must say that I don't "dig" co-location that much.
The point of rebuttals should not be to give the authors a "second shot", but instead to force the reviewers to do a good job. If the review is bad (or incorrect), because of whatever reason, then the rebuttals give the authors a chance to point this out, to the rest of the PC as much as the original reviewer, thereby forcing the latter to make sure they produce a good review in the first place.
ReplyDeleteGiven that all this can be done electronically, I dont see why it should add more than a day or two to the reviewing process...
I experienced the rebuttal trick only once, as a PC memeber of AAAI. Indeed, it adds only a few days to the review process. As a PC member, you are told that on certain dates you are expected to review rebuttals, and change your reviews if appropriate.
ReplyDeleteSo there is a delay of a few days in making the final decision, but most authors will know from the reviews if they are likely to be in or out. I think this is better than some rejections I received or heard of.
For instance, when a reviewer wrote that the main result in a paper is a *direct* consequence of a result that we mentioned in the introduction, we read this to say "you are morons". We wrote a note explaining why it is not a consequence, and asked the PC chair to forward it to him/her.
But in less extreme cases, of course you don't do that.
Hmm, the tone of the post and Eldar's comment seems to be "Computer Science conferences have a lot of problems. Here are some suggestions. None of these apply to us." Is there anything we can do to improve TCS conferences?
ReplyDeleteI'm surprised that everyone rejects the idea of hierarchical/larger PCs so quickly. You don't think that 60-80 papers for a STOC/FOCS/SODA PC member is on the high side? I'm often on PCs in other communities, where 10-15 papers is the norm. This feels like a large enough sample to get a good idea of submitted papers, and to review them. I would have to think hard and possibly turn down hardcore TCS PC invitations because of the large amount of additional work it entails. Plus, reducing the burden on TCS PCs might do something to address the frustration of "We regret that your paper was not accepted. Here are comments from the reviewers: [null]".
If we go to hierarchies this may also mean MORE people on the
ReplyDeleteprogram committee. Many more.
This may make us have to allow
people on the committee to submit.
Currently COLT and ALT do allow
that, but I don't think any other
theory conference does.
This raises the question- should
people on the Program Comm be
allowed to submit. I think NO.
(disclosure- I was on the ALT
committee and DID submit.
While it was done honestly, it
was uncomfortable.)
bill gasarch
My first paper was rejected from Complexity but later accepted at a less prestigious conference. For Complexity one reviewer thought that one of the theorems was false. In fact, this theorem (if he had read the intro) had been concurrently proven by someone else!
ReplyDeleteAs I have basically one data point, I must suggest that perhaps a rebuttal of some sort may be appropriate. It would have taken minutes to point out to this person that he had misread one of the definitions--which was actually rather standard in the field and on which half the paper hinged--and thus his comments made zero sense.
Whether this would have changed his mind, I don’t know (he probably didn’t read the paper anyhow). But it would have made me feel a lot better.
How frequently does this happen? By the number of people suggesting rebuttals, it seems more than my one time; but, like I said, I only have one datapiont.
\begin{restrained rant} The reviewer must have both spent no time and been not been particularly qualified. Why he felt obliged to write anything I don't know? In addition to perhaps getting my paper into the conference, such a rebuttal would have indicated to this person that he should be much more careful reviewing his next article (or just decline the invitation to review).
The incident has left a rather bitter taste in the mouth of this young researcher.
\end{restrained rant}
PS: Answer to “How frequent does this happen?” We would know if we tried.
Bill G. — I'm not sure how much you'd think of it as a theory conference, but Graph Drawing allows PC member submissions without limit. It's a small enough community (and with too few similar alternative venues) that to do otherwise would block too many papers.
ReplyDeleteBut also GD uses web PC software that makes it easy to prevent the submitter seeing any discussion of their submission, at least up to the point at which the total number of accepted papers is discussed. I think it would be more difficult in a conference like FOCS/STOC with physical PC meetings; but then, a larger hierarchical committee wouldn't work so well with those either.
To answer more directly Lance's points: I agree with the 20-30% acceptance range, or maybe more like 25-35%. Anything less than 20% acceptance rate tends to turn the acceptance process into too much of a fashion show and popularity contest, while the times I've been on PCs with acceptance rates more like 40% I've felt that we were accepting too many papers that weren't really advancing the field. Those rates do lead to some difficulty justifying the quality of our conferences compared to areas of CS where the major conferences have 10% acceptance rates, but in many cases I'm not convinced that those conferences are any better than FOCS/STOC, they just get more easy rejections. So I'd prefer to stick with rates that are good for our community and justify our quality in other ways.
Rebuttals: mostly a waste of time and emotional energy.
Colocation: has never worked very well for SoCG at FCRC: higher rates for fewer amenities, less interesting location, more hassle, less interaction with other attendees of the same conference. On the other hand I went to ESA in Europe a year or two back and enjoyed the colocation of with several smaller algorithms workshops, as it led to a broader selection of interesting talks for me to go to. Maybe the difference is that the colocated conferences all still had a common algorithmic focus instead of FCRC's broader all-of-CS approach? I don't know.
I don't really have strong feelings on the rest, but I do feel that the workload of STOC/FOCS/SODA PC members has become overlarge.
The overall acceptance rate is a meaningless figure (unless it is used to compare consequent conferences of the same series). A more useful figure is the number of accepted papers / the number of acceptable papers, because the amount of garbage may vary a lot. That is, the acceptance rate for the second phase.
ReplyDeleteMake PC-s larger. Anything than 15 papers per PC member makes the average quality of reviews subzero.
ReplyDeleteTo make it work, allow PC members to submit one paper. Or allow to submit more but restrict the number of accepted papers to one.
To make this work, have anonymous submissions. There is a variety of web-based reviewing softwares (crypto community has written at least three decent ones), and in the physical PC meetings you can just politely ask the PC member-submitter to leave the room until his or her paper is discussed. (Of course at this moment other PC members would get to know who's at least one coauthor of this paper, but you can do it at the latest possible moment.)
The above has worked very well in Crypto/Eurocrypt and I'm a big proponent of those practices.
Additionally, I like author rebuttals. There are too many bad reviews even at top conferences, so you should deserve a chance to defend yourself. Plus 'submit at the next conference' is not always a good option, especially for say a graduating PhD student.
Additionally, I think top crypto conferences are already getting too selective. I am currently at the KDD 2006 (the top data mining conference) and I like their practice of also accepting short papers. Say you accept $n$ papers now. Some of them (at least 20%) are usually as good as the top $0.2n$ rejected papers. Instead accept $0.8n$ full papers, and $0.4n$ short papers. With stricter page limits, and say 10 min presentation time. And in official bibliographies they have a tag 'short paper'.
Competitions. For example the Electronic Commerce conference runs competitions on various automated trading strategies. Not particularly applicable to theory conferences.
ReplyDeleteHow about a proof cleaning contest? Pick a well-known result with a horid proof and have a competition to see who can clean it up the best.
In case it was not clear from my previous post, actually I don't think that the state of theoretical CS conferences is so bad.
ReplyDeleteOne thing that I would change is to give the PC members leeway to pass on more papers to external referees. This should address several of the refereeing issues that were raised above.
The problem is that conferences count for so much in CS. It is simply not realistic to expect serious refereeing from a conference PC.
ReplyDeleteOur conference system works quite well. We shouldn't try to fix it. In particular, I strongly oppose all of the ideas discussed in the original post, with the one exception of tracking reviews. That's already in practice in FOCS/STOC/SODA, where it is customary to have one PC member that was also on the PC of the previous conference, who often discloses previous discussions on re-submitted papers. (More precisely, FOCS has a previous STOC PC member, and STOC and SODA have a previous FOCS PC member.)
ReplyDeleteWhy would anyone be against increasing the PC so as to reduce the workload? Is it because theory likes to be an exclusionary community?
ReplyDeleteWhy would anyone be against increasing the PC so as to reduce the workload?
ReplyDeletePC's have already increased in size. Typical recent FOCS/STOC PC's have been around 20 people and the 2006 FOCS PC is 23 people vs 16-17 people typical of 5 years ago.
Larger PCs than this would force a major change in how PCs operate:
There is a limit to the number of people whom a conference can support financially to attend a PC meeting and a limit to the number of people you can get in a room to work productively together. Moreover, once you prohibit PC member submissions, as theory conferences do (a good thing), every extra PC member reduces the number of new results that can be submitted so there is an advantage to the community not to have too many people on the PC.
Our conference system works quite well. We shouldn't try to fix it.
ReplyDeleteA system that places so much importance on quick and dirty conference reviews is flawed. At the very least, we should make sure more time is spent reviewing each paper. A system such as AAAI in which each PC member has to find external mini-pc-members who review a maximum of six papers seems preferable.