- Should conferences have double-blind referereing? This was brought up by ***SORELLE*** who thinks that double-blind is good, and later followed up by ***LANCE*** who thinks that double-blind is not needed. I think conferences should have it to avoid bias and perceived bias, so I agree with ***SORELLE***. But that is not the point I want to make.
- Should PC members be allowed to submit? This was discussed in this blog. 10 years ago I thought NO but over time I've changed my mind---it seems counterproductive to not allow some of the best people in our field to submit. But that is not the point I want to make.
- Should these issues be brought up, discussed, with an actual chance of being CHANGED at various business meetings? I STRONGLY think so. Whenever these items are brought up they are shouted down without any real debate. (They are rarely brought up in the first place.) I imagine the following story: Alice goes to CRYPTO and someone brings up stopping doing double-blind submissions. Alice is among the people who shout it down giving the usual arguments against the change, but the real argument is because we've always done it this way. Later in the year Alice goes to STOC. Someone brings up having double-blind submissions. Alice is among the people who shout it down giving the usual arguments against the change, but the real argument is because we've always done it this way.
SO, my strong opinion is that these possible changes should be considered seriously. There must be some change we can believe in! Even if it goes a way I disagree with (e.g., COLT no longer allowing PC members to submit) I would be happy to see that change is possible.
IMHO, changing something just for changing something is irrational. It is completely normal for systems to stabilize as time passes. "Have we changed enough to need a change in the system?"
ReplyDeleteIMHO, changing something just for changing something is irrational. It is completely normal for systems to stabilize as time passes. "Have we changed enough to need a change in the system?"
ReplyDeleteFirstly, Bill clearly isn't saying change should just happen willy nilly. He's just saying that change should be *possible*.
Second, something should only stabilize if the environment it is in has stabilized. It seems pretty bold to claim that the CS Theory community has stablized to such an extent.
My problem with Bill's posts is that he uses far too much capital letters - to the extent that it looks like a graffiti not a respectable post.
ReplyDeleteFor instance, how does this look like:
MY problem with ***BILL'S*** posts is that HE uses FAR TOO MUCH CAPITAL LETTERS - to the extent that it looks like a GRAFFITI and NOT a respectable post.
at lst bll dsnt pst lk ths! u shld b gr8fl!
ReplyDeleteOf course change is possible. There's no secret cabal thwarting the will of the people.
ReplyDeleteThere's always a difficult balancing act: sometimes a determined minority is right and the majority's apathy should not be allowed to stop them, but if you give in to people just because they won't stop harassing everyone else until they are satisfied, it creates bad incentives for future negotiations.
P.S. UN-altered REPRODUCTION and DISSEMINATION of this IMPORTANT Information is ENCOURAGED.
The argument for double-blind reviewing is that it reduced bias, or at least the perception of bias.
ReplyDeleteBut the argument for forbidding PC members to submit is also that it reduces bias, or at least the perception of bias. And I think the bias argument is quite a bit stronger here. You said it yourself, Bill--you assume that are "some of the best people in the field", but these are also the people making the decisions. In this case, the conflict of interest seems VERY clear. (Would this bias be reduced by double-blinding? Of course. Would it be eliminated? No way. And would the perception of bias be reduced? Hell no.)
But yes, these issues should be seriously discussed at business meetings (and more importantly, since business meetings are already too long, elsewhere).
...or KILL me!
Not letting PC member submit the papers has externality on the coauthors. The person choosing to become the PC member is getting something in return for delaying the paper to the next conference, i.e., honor to serve the PC.
ReplyDeleteBut the delay is unfair to coauthors. 20 PC members may roughly amount to unfairness to 30-40 coauthors. The amount of unfairness, due to perceived conflict of interest in case PC member are allowed to submit the paper, is likely to be smaller.
So in principle, not letting PC members to submit the paper brings an unneccessary externality to the collaboration among the community members. In principle, it could potentially harm a student's career who is planning to look for a job, in case his/her coauthor chooses to be on a committee. A delayed paper or a paper published to a second choice conference, in some cases could be a disadvantage in the job market.
Of course change is possible. There's no secret cabal thwarting the will of the people.
ReplyDeleteYou are missing Bill's point. He's not claiming there is a secret cabal. All he is pointing out is that there is too much inertia built into the conference system.
For example, the number of papers in STOC/FOCS as percentage of the TCS yearly paper output has decreased steadily and dramatically since the 80s. My guess is that to have a paper in STOC/FOCS in 1983 it needed to be somewhere in the top 20-30% of all TCS papers. Now the percentage is closer to 3-5%.
There was no conscious decision for this to happen nor were its consequences on the conference themselves and on the field ever considered. It happened because of inertia, pure and simple.
He's not claiming there is a secret cabal. All he is pointing out is that there is too much inertia built into the conference system.
ReplyDeleteI disagree: the inertia isn't a bad thing or even an unintended consequence of other factors. It just means most people don't care, not out of apathy but because they think things are going pretty well and don't see a reason to mess with it. If they cared (and agreed with each other), nothing could stop them. If you want change, make people care. Complaining that change is blocked by inertia is tantamount to complaining about the unfairness of having to take into account the opinions of the people who aren't excited about change.
There was no conscious decision for this to happen nor were its consequences on the conference themselves and on the field ever considered.
Incidentally, this actually has been debated quite a bit over the years. You're right that it wasn't a conscious plan from the beginning, but it has been deliberately allowed to happen despite the facts that people were aware it was happening and that the conferences could have been increased in size. I don't think it is just structural inertia. Instead, it's partly because some people like the prestige that comes with increasing selectivity, and (more importantly) because many people greatly dislike the idea of substantially increasing the size of FOCS and STOC.
It just means most people don't care, not out of apathy but because they think things are going pretty well and don't see a reason to mess with it. If they cared (and agreed with each other), nothing could stop them.
ReplyDeleteIt might also mean that a subset of the people break off from the community and form their own community. There are many such examples : the CG community, the COLT community.
It just means most people don't care, not out of apathy but because they think things are going pretty well and don't see a reason to mess with it.
ReplyDeleteSo the people who get their papers accepted think things are going pretty well, quelle surprise. As such all issues brought up die in the business meeting.
Clayton M. Christensen in his famous book "The Innovators Dilemma" describes various examples of long-gone firms who kept polling their regular customers asking for feedback. Guess what? regular customers are satisfied, that is why they are there. Those firms, Christensen points out, should have polled their non-customers to see what they were doing wrong.
As anonymous #10 points out, how come SoCG, COLT and CRYPTO walked away?
Other questions to ask of the community including both attendees and non-attendees are: Why is non-author attendance seemingly so much higher for SODA than for FOCS? Is the field better off now that STOC/FOCS accept a much more selective set of papers yes/no/maybe? Should we have double blind reviewing? Should we move to short two paper abstracts, ten page submissions, and compulsory full versions in electronic form?
Those firms, Christensen points out, should have polled their non-customers to see what they were doing wrong.
ReplyDeleteThe business scenario is a little different from the academic one. Businesses are trying to maximize the number of customers, and when a business suffers some people necessarily suffer (such as employees whose fortunes are tied to that business). By contrast, CS conferences are not trying to maximize attendance or number of papers. They aren't even trying to maximize research quality or impact outright (but rather to maximize it subject to the constraint that those papers should appeal to the audience). It's also not clear that any researchers suffer very much when new conferences eclipse old ones.
So when a conference has settled on a target audience and approach, and when the audience is happy with how things are going, I don't see a need to change just because the audience and approach could have been different.
As anonymous #10 points out, how come SoCG, COLT and CRYPTO walked away?
This seems like the default behavior to me. As subfields develop and grow, they will eventually have their own conferences. At that point, why shouldn't the best papers move to that conference as well? After all, that's where the community that wants to hear about this work is. I don't think any other solution is feasible. My impression is that the primary arguments against splitting off lots of conferences are twofold. First, there's the belief that we would all benefit from attending incredibly diverse, high-level conferences with most papers unrelated to our own work; there's something to be said for this but it's unrealistic. Second, some people would like to keep FOCS/STOC 90% the same as they are now, with 10% being the very best papers from SoCG, Crypto, etc. In other words, other communities show obeisance to the FOCS/STOC community by admitting that only their very best papers reach the FOCS/STOC standard and that the authors of those papers should feel privileged to attend and mingle with their betters. This is just not going to happen.
Long ago in the past, the whole theoretical CS community was small and a handful of conferences could legitimately claim to contain the very best work going on in all subfields. That's no longer the case. Instead of fighting over who controls the remnants, we should embrace a diversity of conferences and approaches.
So when a conference has settled on a target audience and approach, and when the audience is happy with how things are going,
ReplyDeleteSay, with the current polling system how would you even know if half of the "settled target and audience" has walked away?
This seems like the default behavior to me.
STOC/FOCS claim to be conferences carrying the best of TCS. If some fields walk away then they no longer are so. Hence either we all agree that STOC/FOCS are no longer the end-all/be-all of TCS and as such no longer expect the best people to publish in them or we still believe they are and we need to bring those communities back. In either case, that requires change from the community.
Long ago in the past, the whole theoretical CS community was small and a handful of conferences could legitimately claim to contain the very best work going on in all subfields. That's no longer the case.
You think so and some peple might agree, but as a whole we certainly don't act that way. Go back in this blog and read the many postings where people equate STOC/FOCS with "best of TCS".
STOC/FOCS claim to be conferences carrying the best of TCS. If some fields walk away then they no longer are so.
ReplyDeleteExactly, and this has already happened several times. It's a fait accompli.
You think so and some people might agree, but as a whole we certainly don't act that way. Go back in this blog and read the many postings where people equate STOC/FOCS with "best of TCS".
It's no surprise that those who publish regularly in STOC/FOCS often loudly proclaim that these conferences contain the best of TCS. They've got a lot to gain from encouraging this belief, and it's easy to say with a straight face since it used to be true. Plus they can even regard it as definitionally true if they decide to consider all papers in non-FOCS/STOC areas as inferior by their very choice of topic.
However, practically nobody at SoCG (to name one example) believes it, and even the most diehard FOCS/STOC people have to believe that it is growing less true with time. Right now the real debate is over whether it should be true, not whether it still is, and the "should be" side is losing since simply declaring that the best papers ought to be published in FOCS/STOC doesn't bring them back.
Lets have change, lets ask people attending conference which paper they prefer to be presented. :)
ReplyDeletehow come ..COLT and CRYPTO walked away?
ReplyDeleteThis is inaccurate.
CRYPTO never was contained in the STOC/FOCS sphere and always involved cryptanalysts and practitioners. It has existed since the relatively early days of STOC. Moreover, crypto papers figure prominently in the best paper awards at recent FOCS/STOC conferences so it has hardly gone away.
Early conferences that began as part of STOC were POPL and PODS, though there were specific groups of practitioners who drew them away. Since the mid 1980's several other conferences that remain entirely contained within the STOC/FOCS sphere of topics have been created. (Some of this was from growth of the community, some from the decline of funding for people in North America to attend ICALP, which had been the only third option for theory papers.) The larger theory conferences (that date from the 80's through early 90's) are PODC, LICS, CCC, SoCG, SPAA, SODA, COLT. PODC, COLT, and SPAA all spun off for reasons of connecting with a separate community that had never been part of FOCS/STOC. The sphere of FOCS/STOC work still encompasses all of CCC and almost all of SODA, neither of which has "walked away".
LICS is a clear case of a community that walked away from STOC/FOCS and never looked back but they also focused on reforging some of the connections that had been lost with the STOC/POPL split years earlier.
In principle the topics of SoCG fit entirely within the current conceptions of FOCS/STOC but the presence of this work at STOC/FOCS is small. This seems like the one piece of the STOC/FOCS community that has "walked away" without some other community to attract it.