Logistically physical meetings have several disadvantages.
- Must somehow cover travel costs of participants. Often this comes out of the conference budget, raising registration rates.
- Since traveling overseas is difficult just for a committee meeting, physical PCs have less international participation.
- Invariably some PC member misses the meeting and has substantially less influence on the choices.
The dynamics of physical and electronic meetings differ. Both start by quickly accepting the strongest papers and rejecting the weakest. In a physical meeting the remaining papers get discussed serially. There is a tendency to accept more often at the beginning, realizing you accept too much and start being more stringent and bouncing back the other way towards the end. One also spends too much time talking about the first set of papers, leading to rushed discussions later on.
In an electronic meeting the discussion on all papers happens in parallel. When discussion seems to go a certain way, the PC chair will suggest acceptance and rejection and sees if someone objects. A vote is taken for a few contentious papers. But many papers have nobody who loves or hates them. For those the PC chair has tremendous power for no one will object to whatever they recommend.
Both types of PCs suffer from groupthink, a tendency for groups to reinforce viewpoints. A PC chair also has to make sure that everyone participates in the discussions.
My preference goes for a third approach that I have seen used in a few conferences. The PC members send their reviews to the PC chair who, after some emails for clarifications, makes all the final decisions by him or herself. Simple and, with a good PC chair, works surprisingly well.
Hmm, your 'third option' sounds a lot like the idea of a "benevolent dictator". It rather lives or dies by the quality of the PC chair. Although, if it is easy to set a score threshold so that after doing the easy accepts and easy rejects only a small number of borderline cases remain, maybe this is reasonable. Else, you could go towards a 2-tier PC: many reviewers, a smaller number of "area chairs" who superintend the process and have a PC meeting (eletronic or physical) to sort out the final list. ICDE does something like this.
ReplyDeleteGraham
It seems that conference papers in theory are often not complete, so it would not be easy to judge whether a paper is correct.
ReplyDeleteSo is it the case that the reputation of the authors is very important? Is this a system that is based on trust? Should a paper always have at least one world famous author for a chance of acceptance to a top theory conference?
Hi,
ReplyDeleteA few more advantages of electronic meetings:
- the discussion is recorded in an electronic form. As a result, it is easy to forward the relevant comments to the authors. No one really records the discussions during the physical PC meetings, so it can (and does) happen that comments bear no resemblance to the main issues discussed.
- the discussion happens over an extended period of time, with little time pressure. This promotes careful and independent thinking to much larger extent than physical meetings do. I.e., groupthink is less of an issue (I think).
This said, I do like the physical meeting experience. It is not clear to me if it leads to better decision making though.
Piotr
IMHO a dictator would work well for small focused conferences but less well for STOC-sized general ones. Still the issues with other PC options are valid, so I'm not sure how dictatorial benefits can be scaled up. Perhaps appoint about 3 area-specific "vice-dictators"?
ReplyDeleteBy the way, I think that a dictator would provide a lot of benefit in a physical meeting. Mainly, indeed to cut short a discussion on a paper that starts spiraling down with diminishing returns. Not much different than what I've experienced in other kinds of assemblies.
I find physical meetings to be far superior to electronic meetings. Electronic meetings consume more time, PC members keep switching context between the meeting and their other daily activities, discussions tend to be limited to the people that were assigned to each paper, and usually the only person keeping track of the global picture is the PC chair.
ReplyDeleteWith laptops, wireless access, and publications available on-line, claimed facts can be checked now during the meeting, so the only advantage I used to see in an e-meeting no longer exists.
I very much like the "benevolent dictator" approach. Much more efficient. In democratic decisions by committee, half of one's energy is devoted to making up one's mind about the right decision, but then the other half is spent trying to convince other committee members. With a trusted dictator (who would change from year to year to prevent abuse of influence), we would save an enormous amount of work.
ReplyDeleteI remember one conference where, after all the on-line discussion (there was no physical PC meeting), the 30 accepted papers were almost exactly the 30 papers with highest average score at the beginning of the process. I wonder how common this phenomenon is.
ReplyDeleteAn absolute choice between an electronic vs physical PC meeting vs benevolent dictator does not make sense and the distinction between them is not sharp.
ReplyDeleteFirst, the preferred method should depend on the type of conference. Electronic-only meetings work well for conferences like CCC that have the following two properties:
(1) the conference is sufficiently focussed that every PC member can be expected to understand and be able to have an informed opinion about almost every submission (even though they only review a subset of papers at the start)
(2) the PC members have a level of commitment to the conference that causes them to participate in the discussion after their initial reviews are in.
The benevolent dictator model works best when (2) fails to be true, that is, when PC members are not all that committed to the conference. Unless (2) is satisfied, an electronic-only PC will tend to devolve into the benevolent dictator model anyway, with maybe a small minority of people participating after the first reviews.
However, when (2) is true but (1) is false, i.e., you have a diverse conference with a high level of commitment, and (3) the conference is highly selective, the current form of physical PC meeting works best. Today's physical PC meetings are preceded by extensive electronic interaction among those assigned individual papers. However, with a highly diverse group on a PC there is no good way to compare papers seen by different sets of reviewers unless you get a broad discussion involving the whole PC.
I have been on PCs for FOCS/STOC conferences which satisfy (2) and (3) but not (1) with different PC types: a pure physical meeting, a pure electronic meeting, and the current mixed electronic->physical PC meeting.
In the pure physical meeting the initial reviews were sometimes wildly off the final decision and we struggled dealing with what appeared to be strong papers that had bugs in them - I felt that quite a few decisions were based on way too little information but I did feel that I had understood the decisions we had made.
In the electronic-only meeting, some reviewers uploaded their reviews and simply disappeared leaving strong positive or negative reviews for papers and no way to get further explanation. In the final electronic discussions there was never a time when everyone could listen to the same discussion. Many of the decisions on borderline papers were made without participation of most of the PC. There simply was not enough time for the interaction required to have the experts on a paper justify the paper to the rest of the PC. When we were done I really did not understand many of the decisions that we had made. (Note how different this would be if (1) were true.)
The current electronic->physical PC meetings have worked the best. The electronic part ensures that concerns get ferreted out prior to the physical meeting and the experts can gain some consensus (or determine the basis for a disagreement). The physical meeting ensures that multi-round interaction can take place in discussing papers, particularly borderline ones. I understood the decisions we made and felt there was a pretty good basis for almost all of them even if I didn't always agree.
CS should do what the rest of the world does: Use conference to communicate and meet while leaving the refereeing to journals.
ReplyDeleteThe dictator approach is obviously used in the selection process for journal publication. It seems to me that journals make a pretty bad job in evaluating quality. It's a shame what incredibly weak papers sometimes appear in so called top journals
ReplyDelete(almost never happens with STOC/FOCS etc.). But maybe the dictator is often not benevolent in this case...
It is also a shame that papers
ReplyDeletewith bugs appear in STOC/FOCS
from time to time
(seldom happens in top journals).
I agree that more weak papers tend to appear in journals, but disagree with the sentiment that this never happens at STOC/FOCS. It happens all the time.
ReplyDeleteThe top journals (JACM and SICOMP) hardly ever publish weak papers.
ReplyDeleteNot all papers published there are significantly better than what's published in other journals, but it's always of good quality.
It also very rare that papers with significant mistakes get published in these journals.
Why does it seem that the journal cycle typically takes a year or more (and often two or three years) between initial submission and the submission of the final version (let alone its actual publication)?
ReplyDeleteHi all, I enjoyed very much the experiences with physical meetings and learned a lot about myself and how to argue, and personally I miss them. Still, I keep thinking that the expenses incurred in a physical meeting are too high for just a moderate improvement of the selection process.
ReplyDeleteI have never been to a physical meeting with previous strong electronic interaction like Paul describes, but easily believe this is best for the decisions.
I have been also at electronic two-tier scheme PCs (members/areachairs, for ICML) and found it quite operative, it leads to something similar to property (1), within the areas, and reasonably close to property (2), with respect to the area chairs.
I always thought of an alternative, not explored to my knowledge, which I believe would give the best balance of good selection at fair price: a Very Small group (say, a triunvirate, or tetra or penta) acting the role of the dictator, simultaneously for, say, an hour, on the phone (or videoconf), so that they are focused and listening to the same discussion, AFTER an electronic evaluation and discussion process by a standard PC. It is a scheme rather similar to the other proposals but with slight, but significant, differences...