tag:blogger.com,1999:blog-3722233.post110179038123699163..comments2024-03-28T18:17:00.135-05:00Comments on Computational Complexity: Suggestions to Program CommitteesLance Fortnowhttp://www.blogger.com/profile/06752030912874378610noreply@blogger.comBlogger11125tag:blogger.com,1999:blog-3722233.post-1101995328617917522004-12-02T07:48:00.000-06:002004-12-02T07:48:00.000-06:00Boaz said: "The primary task of the PC is to const...Boaz said: "The primary task of the PC is to construct <br />a program for the conference which will contribute <br />to the conference's audience and to the <br />general community"<br />If this is the case, <br />then the ability to present a paper in public <br />should play a very important role.<br />And I do not think that is good...Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101973465010201352004-12-02T01:44:00.000-06:002004-12-02T01:44:00.000-06:00Another debatable issue is that of "comments to au...Another debatable issue is that of "comments to authors". Oded thinks these should include only "constructive criticism" and any conceptual or technical evaluation of the work has no place here. The other extreme is conferences like INFOCOM where authors have a chance to provide rebuttal to reviews.<br /><br />While Oded's suggestion presumably reduces the work for the reviewers and the PC, I think this doesn't help the community at large. If a paper is rejected, the authors deserve to know what was wrong with it. This has an impact not just on what they do with this particular result, but potentially on their general research direction. This is particularly true for students and all the more so for those outside the top few departments. I think a note "we rejected your paper; don't ask why" is much more annoying than "we rejected your paper because of this".Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101868747341129152004-11-30T20:39:00.001-06:002004-11-30T20:39:00.001-06:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101868760423990322004-11-30T20:39:00.000-06:002004-11-30T20:39:00.000-06:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101868720756601652004-11-30T20:38:00.001-06:002004-11-30T20:38:00.001-06:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101868711682057492004-11-30T20:38:00.000-06:002004-11-30T20:38:00.000-06:00I disagree with the stance that "leading to furthe...I disagree with the stance that "leading to further research" takes precedence over "deserving results". The conferences are mainly for our own community, but yet I feel that conference papers should be judged first by our cause of existence, getting at the knowledge, and only secondly by our means, spending time in research. As an extreme example, I would take the most humdrum "P=NP" paper over the most intriguing "Survey of open problems" paper any day.<br /><br />As for the question of contacting the author of a submission, my take would be to do it only in the case where the correctness of the proof is at stake (as opposed to say the impact of the result), and even then not in the individual PC member level, i.e. decide on this procedure only by email consensus of several PC members or by consultation with the PC chair.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101865192216621932004-11-30T19:39:00.000-06:002004-11-30T19:39:00.000-06:00...continuing my post...
Intentions are much more......continuing my post...<br /><br />Intentions are much more revealing than some absolute quality juggements. Even if a PC member believes that a paper is above some quality threshold that may not relate to whether the paper ought to be presented at the given conference; in evaluating a paper the most useful numerical rating from a PC member to communicate to the rest of the PC (along with the confidence of their assessment) are:<br />* Would they champion the paper?<br />* Are they on the fence about whether to champion the paper?<br />* Would they be comfortable with accepting the paper if other PC members championed it but definitely not going to champion it themselves?<br />* Would they argue against the paper?<br /><br />and for conferences with best paper awards:<br /><br />* Would they argue that the paper should get a best paper award? (Maybe this is a better interpretation than 'I would resign'.)<br /><br />So much for ratings...<br /><br />To believe that extra information about a paper can be ruled out is to fool oneself about the reviewing process. Sometimes there is informal information about a paper long before it has been submitted. Those who are well-connected will always have an advantage in this regard. There will never be a level playing field and it is probably that much more important to clarify issues with people who are NOT in the well-connected circles. <br /><br />If there really is a technical concern with a paper that has a chance of being resolved, then the PC ought to try to resolve it earlier rather than waiting until a final decision several weeks later.<br /><br />The PC should have full information about the extent to which these out-of-band communications have been needed and can make an informed judgement about how this should impact the decision on a given paper. We have to trust the integrity of our PCs to use this information wisely.<br /><br />Paul BeameAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101862945782180002004-11-30T19:02:00.000-06:002004-11-30T19:02:00.000-06:00While I don't completely agree with Oded's scale
(...While I don't completely agree with Oded's scale<br />(I've observed on PCs that the cutoff is more often often around the 6.5 value rather than 5.5 value) the basic principal of the rating scale seems right to me:<br /> The ratings should reflect the PC member's likely <br />votes/actions rather than some abstract notion of quality.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101848706341337022004-11-30T15:05:00.000-06:002004-11-30T15:05:00.000-06:00You might not resign over a paper, but would you a...You might not resign over a paper, but would you at least "fight fiecefully"?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101839425224589982004-11-30T12:30:00.000-06:002004-11-30T12:30:00.000-06:00Actually, actively seeking clarifications is faire...Actually, actively seeking clarifications is fairer than passivley getting the information - you are much more likely to get additional information about results<br />of people you are close to, while you can contact any author for clarification, even if you never met before.<br /><br />In my opinion, considerations such as "fairness" and "deservedness", while important, should NOT be the primary considerations of the PC. <br /><br />The primary task of the PC is NOT to rank the papers according to merit and award the most deserving authors with acceptance to the conference. Rather, The primary task of the PC is to construct a program for the conference which will contribute to the conference's audience and to the general community.<br /><br />That is, the PC is here to serve first its audience and not the authors. For example, the PC can (and should) prefer a paper that is more interesting to the community and may lead to further research over a paper that involves harder and more original work (and hence is more "deserving") but is of less interest to the audience.<br /><br />Getting additional information about a paper may sometimes increase the paper's acceptance chances, and may sometimes decrease them. In any case, it will definitely increase the chance of a higher quality program.<br /><br />--BoazAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3722233.post-1101830121289373732004-11-30T09:55:00.000-06:002004-11-30T09:55:00.000-06:00The best scoring system I have seen was the one us...The best scoring system I have seen was the one used by SIGCOMM.<br /><br />5: Top 5% of submitted papers.<br />4: Top 5%-10% of submitted papers.<br />3: Top 10%-25% of submitted papers.<br />2: Top 25%-50% of submitted papers.<br />1: Bottom 50% of papers.<br /><br />The scores have a clear, understandable meaning. There are fewer buckets, making decisions easier -- and actually, generally much more consistent across reviewers. The system is designed to have you focus on and discuss the papers that might get in, and quickly reject those that are not going to get in. Instead of spending time on arbitrary scoring and averaging issues (Was this a 7 or a 7.5? Is there a meaningful difference between a 6.3 and 6.4 average?) the focus is on determining and dealing with the close decisions. This scoring system helped make the PC one of the best-run I've sat on.<br /><br />SIGCOMM's acceptance rate is now at or below 10%, so I can imagine changing the percentages above depending on the conference. But I'd really like to see theory conferences move away from the much more arbitrary 10 point scale.Michael Mitzenmacherhttps://www.blogger.com/profile/06738274256402616703noreply@blogger.com