The ends of the scale are not so important; a PC does not distinguish between the excellent papers from the merely great nor does it distinguish between the merely bad and the truly lousy but instead a PC must distinguish the pretty good papers from other pretty good papers.
Later on Oded strongly supports using "ex-submission information" in evaluating proposal even to the point of actively contacting authors for clarification during the review process. While PC members do not live in a vacuum and will be exposed to results they have to later review, actively contacting authors is patently unfair as it favors authors with whom you have a working relationship. Authors who cannot give enough information in the submission for the PC to properly evaluate the work deserve to have their paper rejected.
The best scoring system I have seen was the one used by SIGCOMM.
ReplyDelete5: Top 5% of submitted papers.
4: Top 5%-10% of submitted papers.
3: Top 10%-25% of submitted papers.
2: Top 25%-50% of submitted papers.
1: Bottom 50% of papers.
The scores have a clear, understandable meaning. There are fewer buckets, making decisions easier -- and actually, generally much more consistent across reviewers. The system is designed to have you focus on and discuss the papers that might get in, and quickly reject those that are not going to get in. Instead of spending time on arbitrary scoring and averaging issues (Was this a 7 or a 7.5? Is there a meaningful difference between a 6.3 and 6.4 average?) the focus is on determining and dealing with the close decisions. This scoring system helped make the PC one of the best-run I've sat on.
SIGCOMM's acceptance rate is now at or below 10%, so I can imagine changing the percentages above depending on the conference. But I'd really like to see theory conferences move away from the much more arbitrary 10 point scale.
Actually, actively seeking clarifications is fairer than passivley getting the information - you are much more likely to get additional information about results
ReplyDeleteof people you are close to, while you can contact any author for clarification, even if you never met before.
In my opinion, considerations such as "fairness" and "deservedness", while important, should NOT be the primary considerations of the PC.
The primary task of the PC is NOT to rank the papers according to merit and award the most deserving authors with acceptance to the conference. Rather, The primary task of the PC is to construct a program for the conference which will contribute to the conference's audience and to the general community.
That is, the PC is here to serve first its audience and not the authors. For example, the PC can (and should) prefer a paper that is more interesting to the community and may lead to further research over a paper that involves harder and more original work (and hence is more "deserving") but is of less interest to the audience.
Getting additional information about a paper may sometimes increase the paper's acceptance chances, and may sometimes decrease them. In any case, it will definitely increase the chance of a higher quality program.
--Boaz
You might not resign over a paper, but would you at least "fight fiecefully"?
ReplyDeleteWhile I don't completely agree with Oded's scale
ReplyDelete(I've observed on PCs that the cutoff is more often often around the 6.5 value rather than 5.5 value) the basic principal of the rating scale seems right to me:
The ratings should reflect the PC member's likely
votes/actions rather than some abstract notion of quality.
...continuing my post...
ReplyDeleteIntentions are much more revealing than some absolute quality juggements. Even if a PC member believes that a paper is above some quality threshold that may not relate to whether the paper ought to be presented at the given conference; in evaluating a paper the most useful numerical rating from a PC member to communicate to the rest of the PC (along with the confidence of their assessment) are:
* Would they champion the paper?
* Are they on the fence about whether to champion the paper?
* Would they be comfortable with accepting the paper if other PC members championed it but definitely not going to champion it themselves?
* Would they argue against the paper?
and for conferences with best paper awards:
* Would they argue that the paper should get a best paper award? (Maybe this is a better interpretation than 'I would resign'.)
So much for ratings...
To believe that extra information about a paper can be ruled out is to fool oneself about the reviewing process. Sometimes there is informal information about a paper long before it has been submitted. Those who are well-connected will always have an advantage in this regard. There will never be a level playing field and it is probably that much more important to clarify issues with people who are NOT in the well-connected circles.
If there really is a technical concern with a paper that has a chance of being resolved, then the PC ought to try to resolve it earlier rather than waiting until a final decision several weeks later.
The PC should have full information about the extent to which these out-of-band communications have been needed and can make an informed judgement about how this should impact the decision on a given paper. We have to trust the integrity of our PCs to use this information wisely.
Paul Beame
I disagree with the stance that "leading to further research" takes precedence over "deserving results". The conferences are mainly for our own community, but yet I feel that conference papers should be judged first by our cause of existence, getting at the knowledge, and only secondly by our means, spending time in research. As an extreme example, I would take the most humdrum "P=NP" paper over the most intriguing "Survey of open problems" paper any day.
ReplyDeleteAs for the question of contacting the author of a submission, my take would be to do it only in the case where the correctness of the proof is at stake (as opposed to say the impact of the result), and even then not in the individual PC member level, i.e. decide on this procedure only by email consensus of several PC members or by consultation with the PC chair.
This comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteThis comment has been removed by a blog administrator.
ReplyDeleteAnother debatable issue is that of "comments to authors". Oded thinks these should include only "constructive criticism" and any conceptual or technical evaluation of the work has no place here. The other extreme is conferences like INFOCOM where authors have a chance to provide rebuttal to reviews.
ReplyDeleteWhile Oded's suggestion presumably reduces the work for the reviewers and the PC, I think this doesn't help the community at large. If a paper is rejected, the authors deserve to know what was wrong with it. This has an impact not just on what they do with this particular result, but potentially on their general research direction. This is particularly true for students and all the more so for those outside the top few departments. I think a note "we rejected your paper; don't ask why" is much more annoying than "we rejected your paper because of this".
Boaz said: "The primary task of the PC is to construct
ReplyDeletea program for the conference which will contribute
to the conference's audience and to the
general community"
If this is the case,
then the ability to present a paper in public
should play a very important role.
And I do not think that is good...