We just posted the talk schedule for the 2009 ACM Electronic Commerce Conference. Early registration deadline is June 12th and hotel deadline is June 5th.
The EC conference covers a broad range of work connecting computer science and economics. While a large part of the EC community draws from the from the theoretical computer science community, it also draws from AI, computer-human interaction and economists. So the PC process works a little bit differently than in other conferences. Some thoughts.
PC co-Chairs: We had two PC chairs (Pearl Pu and myself) to spread the workload and add some diversity. The workload wasn't that much less but having different viewpoints definitely helped. I worried about what would happen if we had irreconcilable differences but we managed to work around our disagreements. I have nothing but great things to say about working with Pearl, but I would suggest future conferences avoid having PC co-chairs seven time zones apart.
Multilayered Committee: We had a dozen Senior PC members and a very large (about 90) PC members. We let the Senior PC members choose their PC members. Most papers were assigned to two senior PC members who submitted it to two PC members each, some of whom sent it out to outside reviewers. There was discussion between the PC member and sometimes the senior PC members before Pearl and I made the final decisions. The system seemed to work well for this diverse community.
Author Rebuttal: We sent the reviews to the authors before the final decisions and allowed them to give a 500-word rebuttal. Some papers were helped (or hurt) by the author's ability (or inability) to argue against the criticisms given by reviewers. I found the rebuttals quite useful but we did hear some complaints about the extra work required of the authors and the PC.
Would these ideas work for pure theory conferences? I suspect they would work best for the more diverse conferences like STOC/FOCS/ICALP where for any given paper, most of the PC do not have expertise in that paper's area.
You mean, the STOC/FOCS system is not the best? gasp! You mean you are thinking about changing things in STOC/FOCS? gasp!
ReplyDeleteI think the system of hierarchical PC is sub-optimal and so is choosing 90 PC members. Yeah, you could have 90 reviewers, but many of these PC members were looking for external reviewers. So it is 90 PC members + another set of reviewers.
ReplyDeleteSo in effect a reviewer review a paper, then the review climbs three levels of hierarchy.
You did not seem to say anything about the accuracy of the review process. I have been to EC PC, and the accuracy of its review process is less than desirable. Inaccurate in the sense that, the EC process results in higher than desirable level of noise to signal ratio.
I like the author rebuttal opportunity. But given that EC is about the third of a size of stoc/focs, one could have 10 PC members (call them senior or not), and about a 100 reviewers. This might raise the accuracy of EC to the same level as stoc/focs.
I understand that EC allows PCs to submit papers, given 90+ PCs. I cannot imagine anyone else left removing these names.
ReplyDeleteHowever, I don't understand why senior PCs are allowed to submit papers, which is not the case for many other conferences.
I agree with the previous comment that senior PC should not be allow to submit the papers. Senior PC for EC, is basically the same as PC in other theory conferences.
ReplyDeleteEC this year got a new terminology, and diluted the recognition PC, which the theory community has and consider it prestigous. Being a PC member is a presitigous thing, but not anymore, since a reviewer could essentially be called PC, and therefore look for another reviewer.
Ever wonder, why an ISP offers bandwidth in bits per second, and not bytes per second, when we usually measure the file sizes in bytes? Maybe because which ever ISP will offer bandwidth in bytes per seconds, that ISP will look to offer less speed than others. This principal holds in other places too where a terminology giving a diluted impression is adopted as predicted by Nash equilibrium.
If we are not careful, this beaurocracy of senior PC and PC could potentially dilute the term PC. In order to attract the best PC members, other theory conferences may be required to create this distinction of Senior PC and PC.
The theory community is built on the basis of equality. We even list our names on a paper with alphabetical ordering. So whenever we want to change the basis of equality, we should look for an input from the broader theory community.
I agree with you Kamal. I particularly don't like such a large list of PCs of EC. I admit that most AI conferences use this rule, like AAAI and IJCAI. However, do they have a good reputation about it? I don't think so. In theory, it's a kind of honor to serve on PC of STOC/FOCS, where EC turns it down..
ReplyDeleteGiven that the size of the EC PC is about twice the number of accepted papers, isn't it a bigger honor to present a paper than to be on the PC?
ReplyDeleteEvery discussion about STOC/FOCS system seems to ultimately boil down to questions of "prestige". If it is really the case that it is only "prestige" that motivates people to serve on PC, rather than a sense of citizenship and loyalty to the dicipline, then TCS is in big trouble.
ReplyDeleteIt is also not just the question of PC. The motivating factor behind
"publishing" in FOCS/STOC proceedings also seems to be "prestige" more than anything else.
Every discussion about STOC/FOCS system seems to ultimately boil down to questions of "prestige". If it is really the case that it is only "prestige" that motivates people to serve on PC, rather than a sense of citizenship and loyalty to the dicipline, then TCS is in big trouble.There is a fair amount of sacrifice involved in being a member of a major theory conference PC. There is both the time commitment (8 to 10 weeks during which one does not have much time for other pursuits) and the inability of PC members to submit papers. Though there is a lot of value to be learned from meeting other members of the PC and hearing their perspectives, if being chosen as a member of a STOC/FOCS/SODA PC did not come with significant prestige then it would be hard to argue that its benefits would outweigh the costs for young pre-tenure researchers. With the current scheme involving significant prestige, the major conference PCs gain the insight and freshness from young researchers and it works to the benefit of of the young researchers as well.
ReplyDelete... if being chosen as a member of a STOC/FOCS/SODA PC did not come with significant prestige then it would be hard to argue that its benefits would outweigh the costs for young pre-tenure researchers ...
ReplyDeleteIn my opinion young pre-tenure researchers have no business being in the scientific committees of the leading conferences in any area. Their time is better spent on other things -- like learning and doing research. It is just ridiculous that these days we see even postdocs on PC's of CS conferences (even graduate students in non-theory conferences). This is completely out of whack with the standard practices in other scientific disciplines, where similar tasks fall on the senior members of the community. I think the senior researchers in TCS have largely abdicated this crucial responsibility -- leading to shallow reviews, flawed and abitrary processes and decisions etc. that result from these committees stuffed with young researchers who haven't had time to develop the knowledge and perspective needed to evaluate papers and sub-reviews.
I have been on several theory PCs, including a couple of FOCS/STOC/SODA. In all these PCs, the young pre-tenure researchers typically did a much more serious job, gave better and more thought-out reviews, participated in discussions of other papers that were not assigned to them, etc. Senior members, in contrast, tended to give one-line reviews, and argue hard for a couple of papers (in favor or against), but without much technical justification. In fact, the senior members often seemed lazy.
ReplyDeleteIn all these PCs, the young pre-tenure researchers typically did a much more serious job, gave better and more thought-out reviews, participated in discussions of other papers that were not assigned to them, etc.
ReplyDeleteThe young are enthusiastic no doubt -- but are they wise ? I think not (most of the time).
There are several good reasons to have young people in the PC.
ReplyDelete-It shows them how the process works and that there is no mafia or hidden agenda.
-It keeps the field fresh. In other disciplines entrenched interests slow down new developments.
-Young researchers tend to give more thorough reviews. Senior researchers tend to give simple up or down reviews. This is not laziness but a reflection on time availability on their part.
On the other hand, I found that often young researchers had a harder time judging conceptual developments.
"The young are enthusiastic no doubt -- but are they wise ? I think not (most of the time)."
ReplyDeleteWeird comment given that even senior PC tend to subreferee everything. In my experience, senior PC just cut and pasted the subreviewers comments even when they didn't make sense. That is my definition of lazy. Of course, some more junior people did that too.
How can it help if you are wise if you do not know anything about the papers assigned to you? Better to know something than to be wise and not apply the wisdom.
"On the other hand, I found that often young researchers had a harder time judging conceptual developments."
ReplyDeleteWhat exactly do you mean by this comment? That you disagreed with their opinions?
"On the other hand, I found that often young researchers had a harder time judging conceptual developments."
ReplyDeleteWhat exactly do you mean by this comment? That you disagreed with their opinions?
I am not the previous poster but there is a certain conservatism in those who have not experienced the entire cycle go from wacky-sounding idea to fundamental progress in the field. (Who would have thought that multi-prover interactive proofs would have led to the PCP theorem?!)
This is not a matter of a major disagreement but rather a tendency to rely more on the concrete than the conceptual in making the close calls.
"This is not a matter of a major disagreement but rather a tendency to rely more on the concrete than the conceptual in making the close calls."
ReplyDeleteOf course if senior people really thought that incorrect "close calls" were hurting the field, then we (they) should fight to change the status quo and accept more papers! Then people could judge for themselves if an idea was "conceptual" or not. The question is not whether or not junior people are qualified to judge papers, but why do we require that a certain select group of people judge papers?
It's great the multiparty interactive proofs lead to PCPs, but what if there are (or were) other ideas out there that never got a chance?
In all these PCs, the young pre-tenure researchers typically did a much more serious job, gave better and more thought-out reviews, participated in discussions of other papers that were not assigned to them, etc.
ReplyDeleteNo one doubts that younger pre-tenure people are more energetic in the PC -- it is to be expected since it is their first taste of power. However, there is something inherently wrong in a system that depends on young pre-tenured researchers and postdocs to sit in judgement of the works of senior people in the field, and to a certain extent dictate the direction of research. Might be moving away from a conference-based publication/reward system, towards a more mainstream journal based one like in other areas of mathematics (a direction in which the wind has finally started to blow viz. Vardi's recent article) will correct this anomaly.
"However, there is something inherently wrong in a system that depends on young pre-tenured researchers and postdocs to sit in judgement of the works of senior people in the field, and to a certain extent dictate the direction of research."
ReplyDeleteI don't see anything wrong here. Can you be more specific?
Senior people are not inherently better qualified to determine the best direction of research. And we all know examples of senior people who have chosen a direction of research and never changed it.
The system needs some balance on both sides, and that is what we have.