So instead I offer to you this short movie (16 MB, 3:14) using soccer to explain Euclid's theorem that there are infinitely many prime numbers. Part of a new British project science.tv.
So instead I offer to you this short movie (16 MB, 3:14) using soccer to explain Euclid's theorem that there are infinitely many prime numbers. Part of a new British project science.tv.
I watched the movie and could find no connection between the narrative (a nice exposition of Euclid's proof) and the visual (mostly a plain old amateur soccer match). To me it looked a bit like reciting from The Animal Farm to the tune of The Ring Cycle, or is it only me?
ReplyDeleteWow, is the person with the most papers in FOCS this year an undergrad?
ReplyDelete> Wow, is the person with the most
ReplyDelete> papers in FOCS this year an undergrad?
No, Assaf Naor received a BSc from the Hebrew University of Jerusalem in 1996. =)
I think he meant Mihai.
ReplyDeleteThere's a goof with the voice over! He says football instead of soccer. They probably did the voice recording before the clips were even edited.
ReplyDeleteMihai is now a graduate student too, so it doesn't apply either. Still, three FOCS papers for a first year PhD student is not too bad.
ReplyDelete"Mihai is now a graduate student too..."
ReplyDeleteHe wasn't when he submitted.
Eldar,
ReplyDeleteI believe that the only connection is in Marcus du Sautoy's love for football and prime numbers. The team you see playing is indeed his own amateur football team, and the team does play regularly with jerseys sporting prime numbers on.
Marcus du Sautoy is professor of mathematics at Oxford University, and the author of the excellent book
The Music of the Primes. Check it out, if you have not read it already.
Marcus du Sautoy is professor of mathematics at Oxford University, and the author of the excellent book
ReplyDeleteThe Music of the Primes. Check it out, if you have not read it already.
I'll second that emotion!
Nice program. And now, let's give some awards based on the titles.
ReplyDeleteBest titles :
-------------
Planar Earthmover is not in L_1 -- informative and concise
Accidental Algorithms -- very intriguing!
How to Play any Unique Game -- arouses curiosity
Worst title:
------------
Local Peeing and Service Contracts in Strategic Network Formation -- inappropriate for FOCS Theory
Longest title:
--------------
Point Location in o(log n) Time, Voronoi diagrams in o(n log n) time, and Other Transdichotomous Results in Computational Geometry
Most boring titles:
-------------------
(They say what they do, no more and no less)
An Omega(n^{1/3}) Lower Bound for Bilinear Group Based Private Information Retrieval
Faster Algorithms for Approximate Distance Oracles and All-Pairs Small Stretch Paths
Improved Bounds for Online Routing and Packing via a Primal-Dual Approach
Improved approximation algorithms for multidimensional bin packing problems
Most secretive title :
-----------------------------------
Points on Computable Curves -- surely the authors can make a little effort to tell us more!
Words conveying the least information:
-----------------------------------
"Fast" (as in "Fast algorithms" -- when is someone going to advertise "slow algorithms"?)
"Faster"
"and More"
"Efficiently"
Most unappealing word to occur in a title:
-------------------------------
"Improved" -- that's a big turnoff for me. I'd rather read the original or the final ("optimal") construction than an "improved" anything.
Runner-up: "Towards"
"Local Peeing and Service Contracts in Strategic Network Formation -- inappropriate for FOCS Theory"
ReplyDeleteThat's pretty inappropriate, I agree. But that's not the title, fortunately.
Most unappealing word to occur in a title: "Improved" and "Towards"
ReplyDeleteInteresting. Science is built one brick at a time, but if the title happens to acknowledge the incremental nature of what we do, then it is "unappealing".
It seems that papers named "towards" should be the ones with longer lasting impact. They are presumably addressing a problem of such importance and difficulty that even a partial solution or analysis of the problem is worth publishing. In contrast, yet another optimal algorithm for some made up problem is likely to have much less relevance.
Come on, it's unfair to count me as graduate student :-) I'm only starting in September. These were definetely undergrad papers.
ReplyDeleteWhile talking about titles, don't we get some notice for the "most oxymoronic title" or something? :-) We tried pretty hard:
Higher Lower Bounds for Near-Neighbor and Further Rich Problems
Come on, it's unfair to count me as graduate student :-) I'm only starting in September. These were definetely undergrad papers.
ReplyDeleteNah, let's face it: you are now over the hill. :-) :-)
"Come on, it's unfair to count me as graduate student :-) I'm only starting in September. These were definetely undergrad papers."
ReplyDeleteThis is not fair. Only being undergrad is not important. Your age should be around 25 in which some people complete their Ph.D. and even people like Erik Demaine are associate professor at that time. So just being an undergrad is not that important, esp. if you have started research at the edge of 20.
Actually I am still 23; I think it's a bit of a stretch to say this is an age when many people complete PhDs.
ReplyDeleteOf course, I could have gotten much further by now, Erik being the perfect example :-)) On the other hand, it's all in the choice of the metric -- I do currently have more STOC/FOCS papers than him (so no comparison with him when he got the job at MIT).
While at it, let me give you something to ponder. My undergrad career was made possible by Erik, who decided to fund a freshman to work on whatever he fancied (honestly, I had him convinced for maybe 1 year that my main interest is number theory). He continued to give me unconditional support (funds, university rule bending etc), even as it became clear that we are very incompatible researchers and the chances we could actually work together on a problem are virtually nil.
Really, how many professors would put so much faith in an undergrad? It's a style of advising that works for a tiny minority, and it takes a lot of inspiration to get it right.
Higher Lower Bounds for Near-Neighbor and Further Rich Problems
ReplyDeleteHow can that have escaped me. Great title! If you could just keep the boldface for the final title, so that people like me, who are not into subtlety, have a chance to notice it, that would be great...
Titles can still change while authors are preparing the final versions. Maybe we can collectively give them advice to build better FOCS titles?
ReplyDeleteFor example (not knowing the actual paper contents):
Generalized Dobrushin Uniqueness
Local Switching Connects Peer-to-Peer Networks
Inclusion-Exclusion colors graphs in O(2^n)
Non-Uniform Buy-at-Bulk Network Design
Allocation problems: beyond 1-1/e
On the Hardness of Learning Halfspace Intersections
Algorithms for LogConcave Functions
Computing the surface area of a convex body with heat flows
On random projections of matrices
Primal-Dual Routes and Packs Online
On Dynamic Planar Point Location
On Multidimensional binpacking
Inclusion-Exclusion counts set partitions
PageRank Vectors partition graphs locally
Learning Halfspaces and Parities with Noise
Breaking the information theory barrier in computational geometry
Lloyd type methods for k-means
Come on people, everyone knows that the universal formula for calculating TCS coolness is
ReplyDelete(# FOCS/STOC papers)*(number of sexual partners)/(AGE - 20)
(# FOCS/STOC papers)*(number of sexual partners)/(AGE - 20)
ReplyDeleteshouldn't you square the age, if you want to measure some steady rates for #papers/year and #partners/year?
This year MIT is amazing in the number of accepted papers. Is there any one from Cornell or Berkeley (I see at least one from CMU and one from Stanford)?
ReplyDelete"have more STOC/FOCS papers than him"
ReplyDeletePlease do not use again this kind of completely incorrect comparison of couting only STOC/FOCS papers and ignoring SODA/SOCG papers esp. for an algorithms person.
How can a comparison be incorrect? - Maybe its incomplete. So don't forget to count the PODC, SPAA, ESA, ISAAC, RANDOM ... papers esp. for an algorithms person.
ReplyDeletePlease do not use again this kind of completely incorrect comparison of couting only STOC/FOCS papers and ignoring SODA/SOCG papers esp. for an algorithms person.
ReplyDeleteYes, I totally agree. I was merely emphasizing that people impose different notions of success on themselves, eg it's not totally undefendable to have staid for a full 4 years in undergrad and gotten more STOC/FOCS papers in the process :-)
On a different note, this view of "STOC/FOCS and algorithms people" seems mostly an artifact, esp when looking at the current list of accepted papers. Lots of very diverse algorithms in there...
I hope the people commenting above are imposters, otherwise it seems that TCS is full of - sorry to say this - fools.
ReplyDeleteeg it's not totally undefendable to have staid for a full 4 years in undergrad and gotten more STOC/FOCS papers in the process :-)
ReplyDeleteIt is quite Ok to do have taken longer. Just ask Erik how many years earlier he could have finished his PhD if time had been the driving consideration.
this view of "STOC/FOCS and algorithms people" seems mostly an artifact, esp when looking at the current list of accepted papers.
One swallow does not Spring make.
I never heard so much "academic" shit like the above. It seems that education at MIT is at an all time low.
ReplyDeleteSpend less time counting your papers, or how many months will take to do better than somebody else, and go have a life.
"have more STOC/FOCS papers than him"
ReplyDeleteMihai, forget the numbers. Over the long run people won't remember how many papers you have, but rather the quality of the work you do.
Question: how many papers did Einstein published in his life? Answer: who cares!
Come on, don't attack the guy for an unfortunate phrasing.
ReplyDeleteWhat Patrascu has accomplished so far is rather impressive (even if you see his research as a number; ~10 stoc/focs/soda papers).
How many graduate students accompish that by the end of their PhD?
And by the way, numbers *are* important: A PhD graduate with 4-5 stoc/focs papers is considered to be of "top quality" - at least for the job market.
I agree of course that "numbers" is an artificial metric; however, it is arguably the most influential variable when one is on the academic market.
Moreover, how many researchers (especially students) produce top-top work?
Come on, don't attack the guy for an unfortunate phrasing.
ReplyDeleteUnfortunate phrasing? Have you seen Mihai's web page?
Allow me to be honored for you giving me Einstein as an example :-)
ReplyDeleteThe issue is a bit stupid. My usual rhetoric is about too many people treating this as a numbers game (esp given too many publication venues). People who know me personally have probably heard this a few too many times. The only other time I posted to this blog was a few weeks ago, when I was saying exactly this (and yes, I was signing just like now), and of course I got slammed for it.
So no, I'm not one who thinks numbers are too important. But if you do talk numbers, get them right, and don't tell me I'm a quasi-loser for not having a PhD+tenure by this age. Like I said, those criteria are virtually as relative as numbers.
Allow me to be honored for you giving me Einstein as an example :-)
ReplyDeletePermission granted :-)
But if you do talk numbers, get them right, and don't tell me I'm a quasi-loser for not having a PhD+tenure by this age. Like I said, those criteria are virtually as relative as numbers.
That comment was either in jest or by someone jealous of your success. Either way, pay no attention to it.
Unfortunate phrasing? Have you seen Mihai's web page?
ReplyDeleteIt's good the ball was raised to the net about this one :-) I think people should be all but forced to create something like this. Maybe this post will convince more people to do it.
The point is, we have too many publication venues. So what do you think about somebody who publishes 30%+ of papers is STOC/FOCS/SODA? He's a guy with good intentions, but not God, so he of course can't always get perfect results, and likes to also record partial progress in smaller conferences.
What do you think of a guy with 80% of the papers in CCCG/ISAAC/COCOON/...? He's a guy who would only think of an important problem by accident; his usual style is to invent minor things and generate publications.
NB, some people are trying to address this problem in other ways, eg Erik is convincing people to treat CCCG as a technical report, not a conference publication. But that is not solving the issue entirely. If people are just upfront about what they're doing, everyone can apply individual standards (eg, some people think ICALP is a top conference, others think it's not).
"But if you do talk numbers, get them right, and don't tell me I'm a quasi-loser for not having a PhD+tenure by this age."
ReplyDeleteI guess what he meant is that you cannot be considered a "standard" undergraduate student, not because you are 24, but because, when you started at MIT, you already had extensive hard-core training in computer science in general and algorithms in particular.
So, you had a huge advantage wrt your peers and could start doing research right away.
So what do you think about somebody who publishes 30%+ of papers in STOC/FOCS/SODA? He's a guy with good intentions, but not God, so he of course can't always get perfect results
ReplyDeleteI think it is ridiculous you can ask this question on numbers alone. It illustrates your obsession with numbers. Here's a person who has 30% of his papers in STOC/FOCS/SODA: Stephen Cook. Perhaps you've heard of him?
What do you think of a guy with 80% of the papers in CCCG/ISAAC/COCOON/...? He's a guy who would only think of an important problem by accident; his usual style is to invent minor things and generate publications.
Same thing. What are the other 20% of his/her papers? Solutions to major conjectures or technical contributions to the topic du jour? One cannot judge without looking at the papers.
I think it is ridiculous you can ask this question on numbers alone. It illustrates your obsession with numbers. Here's a person who has 30% of his papers in STOC/FOCS/SODA: Stephen Cook. Perhaps you've heard of him?
ReplyDeleteActually, it seems that if you count the right way, and look at the percentage of Cook's conference papers that appear in STOC/FOCS, it is a whopping 77%.
Actually, I think that was supposed to be the good case. Cook would be a "guy with good intentions, but not God"
ReplyDeleteI do not think one should judge the merit of a paper solely by the conference it appears in. There are many FOCS/STOC papers which have been consigned to oblivion once published and there have been numerous papers in the so-called lower ranked conferences which have been widely read and cited.
ReplyDeleteExtremists ruin everything. Let me try to condense this: If I want an indicator which (1) has small complexity and (2) is highly correlated with research quality, then the clear choice IMNSHO is # FOCS/STOC publications/year of research.
ReplyDeleteIs it perfect? Of course not. Should it be used to compare two particular people? No. Is it highly correlated with research quality as I (and many others) would judge it? Yes. Of all measures with comparable correlation, does it have the smallest description? Yes.
Is it useful for anything? Probably not.
Is it highly correlated with research quality as I (and many others) would judge it? Yes
ReplyDeleteNot clear. A lot of people would rather have one really high quality result a year than three mediocre STOC/FOCS publications a year. I don't know about you, but I would consider a person who does the former a more successful researcher.
It seems that either the imposters above do not care for being exposed, or TCS is really full of morons.
ReplyDeleteFOCS/STOC are good conferences for hot subjects in the algorithmic-complexity branches of computer science. That is:
ReplyDelete1. Hot current subjects have clear preference;
2. If a paper is not algorithmic-complexity oriented it will not be accepted due to relevance.
Therefore, FOCS/STOC is a too narrow venue that cannot be taken seriously as a scientific quality measure for TCS.
In order to evaluate scientific achievements in the long run, FOCS/STOC measure is clearly irrelevant.
Of course, I'm not talking about ad-hoc, short term measures, that might be proper for the mediocre level of scientific research. For such ends, these kind of simple instruments to measure quality of research might work.
This year MIT is amazing in the number of accepted papers. Is there any one from Cornell or Berkeley (I see at least one from CMU and one from Stanford)?
ReplyDeleteWhich is the CMU paper? Are you counting Ryan O'Donnell?
Is there any doubt about that a JACM paper is much more prestigious than FOCS/STOC papers? Usually, only the the best paper award winner gets invited to JACM (not into a special issue, just a regular issue). So in this case, just STOC/FOCS is not the only issue.
ReplyDeleteI am curious. What is the difference between a special issue paper and a regular issue paper, say, on SICOMP or JCSS? Generally speaking, which is more prestigious?
ReplyDeleteFun fact: The last three FOCS PC's have not had any MIT members ... it must be some kind of a record.
ReplyDelete"which is more prestigious?"
ReplyDelete"most papers in FOCS.."
"numbers *are* important"
"have more STOC/FOCS papers than him"
It is a great promise for the further advancement of the very serious branch of science called TCS to see that researchers in the field spend most of their energy on the deepest and most important questions.
Guys, keep up the good work!!
then the clear choice IMNSHO is # FOCS/STOC publications/year of research.
ReplyDeleteIt is curious that this post follows a discussion in which it was highlighted that crypto people (for whatever reason) tend not to publish in FOCS/STOCS. Neither do logic and machine learning people, who prefer COLT, and some have argued that similar self-selection takes place with computational geometry and SOCG. I know of people with several STOC/FOCS papers that self-select and submit most of their algorithm papers to SODA for the simple reason that the intended audience regularly attended SODA, but not STOC/FOCS.
Yes, we do need quick and dirty measures of quality to compute promotions, grants, etc. However it is possible to be too quick and too dirty. The h-index is one such shorthand, the STOC/FOCS index is only slightly quicker but it's way dirtier.
Who are the machine learning people who write papers that would be applicable to FOCS/STOC? I get the impression that the mathy ML papers read like statistics papers, not necessarily TCS papers, and the less mathy ones belong in AI journals.
ReplyDeleteMy guess is that the folks who say that what is important is quality, not numbers of pubilcations in FOCS/STOC/Whatever, at least have jobs, and more likely have tenure. I have a hard time telling my PhD students to not worry about so much about numbers, when as best as I can tell, in the short term, numbers are very important.
ReplyDeleteIt's amazing that in such a group of supposedly smart people, the concept of "correlation" is so misunderstood. Your reactions are similar to the idiots who were deeply offended by Bill Bennett's comments last year, when the Guardian ran a story entitled "Abort all black babies and cut crime, says Republican."
ReplyDeleteWhen someone suggests that # of F/S papers is correlated with research quality, it is idiotic to make an objection of the form: "What about Professor A, that would mean he sucks, right?" It is also silly to posit cause and effect (e.g. being male is correlated with research quality as well), and finally such a statement does not endorse any course of action based on this correlation.
In the end, I think the whole point was that there are sects of TCS researchers for which the goal is to publish as much as possible, as often as possible, without ever making (or even trying to make) really significant contributions, and the F/S metric tends to weed those people out.
Who are the machine learning people who write papers that would be applicable to FOCS/STOC?
ReplyDeleteJust about the entire COLT conference, in my opinion. In fact, they used to publish in STOC/FOCS as well but as COLT came on to its own they moved away.
My guess is that the folks who say that what is important is quality,at least have jobs, and more likely have tenure.
ReplyDeleteYou are missing the point. *Do* focus on numbers when planing your short term career goals or your interview goals. *Don't* focus on numbers when doing public comparisons of quality as you only look foolish in the process.
It's amazing that in such a group of supposedly smart people, the concept of "correlation" is so misunderstood.
ReplyDeleteWe understand correlation quite well, thank you very much. The point we are making is that the correlation is too weak to make a blanket statement such "as Mihai is ahead of Erik because he has more F/S papers" as Mihai said tongue in cheek. For an index to be a good measure it has to have high correlation with what is trying to measure. The F/S index mismeasures all cryptographers, logic people (theory B), learning people, algorithms people, and computational geometers. It also overmeasures people who are trend chasers, always publishing irrelevant results in the hot topic du jour.
The F/S index mismeasures all cryptographers, logic people (theory B), learning people, algorithms people, and computational geometers.
ReplyDeleteI disagree with all of these except for logic. Just choosing some favorite cryptographers, e.g. Goldwasser, Goldreich, Micali; they are not underpredicted by F/S counting. The case for (theoretical) learning, algorithms, and computational geometers is even stronger.
I disagree with all of these except for logic. Just choosing some favorite cryptographers, e.g. Goldwasser, Goldreich, Micali; they are not underpredicted by F/S counting.
ReplyDeleteHuh? Micali has a grand total of eight F/S papers in the last sixteen years (1990-2005), or about the same as any of a constellation of moderately accomplished non-cryto researchers. Goldreich has published less than 10% of his papers in F/S in the last ten years (1996-2005), which doesn't even clear half of the 20% threshold suggested by Mihai.
My guess is that the folks who say that what is important is quality, at least have jobs, and more likely have tenure.
ReplyDeleteHere's another option: these are the people who are not getting published in FOCS/STOC.
Instead of facing their incompetence, they romanticize it. "It's not that my work is mediocre, or that I'm slow. I'm just working on the really important problems."
But someone like Mihai forces such people to face reality. Hence, the emotional responses seen in the comments here.
Spend less time flaming Mihai and concentrate on doing excellent research. (Which, let's face it, will get you in FOCS/STOC.)
Goldreich has published less than 10% of his papers in F/S in the last ten years (1996-2005), which doesn't even clear half of the 20% threshold suggested by Mihai.
ReplyDeleteYou're dumb.
In the last 10 years, Oded has published 38% of his conference papers in F/S. By your count, if Oded puts his papers on ECCC and sends them to a journal, then the highest percentage he could achieve is 33%.
Come on. Goemans has 8 papers in F/S and Ravi has 12 papers in F/S in their whole life. By your standards these very top algorithmic people should be almost like Mihai. Doesn't it seem stupid? It is clear that F/S is more for complexity and more generaly lower bounds than algorithms or upper bounds. These people have around 14 SODA papers each.
ReplyDeleteJust a note: Patrascu is not the only author of most of his publications.
ReplyDeleteSo, "top" conferences like FOCS/STOC are worshipped, and any doubt about whether the number of publication there reflect one's ability will be labled as blasphemy..... and blasphemy has a new name in the 21st century: "sour grape". OK, so let's have a little more blasphemy by looking at the following long time doubts I have about the holy peer-review process in these worshipped conferences:
ReplyDelete1) By "top" conferences, we usually mean "general" conferences that cover many areas of a bigger field (like FOCS/STOC). As a result of this generality, submissions to these conferences are possibly reviewed by people of greatly varying background. Ideally, this means the work are evaluated by their value to the entire field, instead of how important it is to a very specific area. However, we can also phrase this as how well can the reviewer understand and appreciate the work, and what is his attitude to people/research in the particular area. Remember that some (if not all, in the worst case) of the reviewers can be non-expert to the specific topic involved. For one thing, experts for particular areas may be of limited supply, and even if there are plenty of them, they may already be assigned too many papers, and I truly have no idea how this assignment is done. Random? Or work of particular "interest" got assigned reviewers that are less likely to return gibberish comments?
Furthermore, I believe work that addresses fundamental issues with a
novel approach, which can have significant, long lasting impact to the field, can only be appreciated by those with profound knowledge of the specific area. And what will the fate of these work be in a "top" conference if
the author is from some school you've never heard of, if the PC chair did not pay attention when assigning reviewers? This paper may got one 4 (or even 5) from one reviewer, and then two 1 or 2 from the non-experts who don't really know what's the big deal about the work, where is the novelty.
The conclusion is, submissions to these "top" conferences may not be
evaluated by how well they fit into the big picture, but how well they fit into the small part of the big picture known to the reviewers assigned. I believe, this is making the fate of a paper submitted to these "top" conferences "random".
2) If you have been in research long enough, I suppose you should have heard of, or even have personal encounters with the "Mafias": powerful schools/individuals who have some say in PCs of some "top" conferences, and who are seen to have underserved advantage in this peer review process. If you haven't, I suggest you ask around, you should get some positive feedback before you ask the 10th person. I think many supposedly suffering from the "sour grape" symptoms will notice, some papers accepted by these top conferences would never be accepted if they
come from somebody studying in some unkownn schools. I have some personal experience with these "Mafias", I know friends who talk about them, and I know people who have been in the PC of "top" conferences admit their existence.
The conclusion to the second point is, the peer review process is random to some, and pretty certain for some others.
If you read all the shit above to this point, I suppose you either agree with me a lot (or you'll have lost patience and skip to the next post), or totally disagree (so you are only reading to pen up a longer message attacking mine). But, as scientists(supposedly ..... didn't we call our departments Computer Sciences?), I challenge you guys to come up with scientific ways of resolving our disputes, and design experiments to verify whether this "random" process that roots from the lack of understanding or appreciation of the reviewers, and this unfair process due to the Mafia exist. To confirm, or refute through experiments, instead of "arguments" the claim that these "top" conferences are good measures the quality of one's work, that good work, and good work alone, will get in. How?
All we need is a repository of the reviews for different rejected
submissions to these top conferences, together with the submitted work. Both should be authenticated (the reviews really come from the PC of the conference involved, the submission is really submitted, and tied to the reviews), and concealed by encryption so that only the author can make the information public when he's willing to (possibly after the work involved is published, or given up). Through these stored, and revealed reviews, we can see whether the unknowledgable reviewers that I have talked of exist. By comparing the rejected work with those accepted by the same conference
at the same year, and applying those reasons for which the rejected got its fate on those accepted, we can see whether the double standard I have talked of exists.
Yes, this is in fact a mechanism to evaluate the well worshipped peer review process. And we do need an evaluation and monitoring mechanism. We should keep in mind that, anything left in the hands of humans, unchecked, unchallenged, will certainly be abused, either for convenience (it's sometimes quite tiring to do the right thing), or for personal benefits. This will require certain extra work by those organizing conferences. But I think people in the community should force it on those organizing conferences, to make sure that their top conferences deserve the reputation, and the "nobody" are not wasting their time so that those "somebody" can put "acceptance rate: 1%" on their publication list.
"Here's another option: these are the people who are not getting published in FOCS/STOC."
ReplyDeleteWhy not face another reality: F/S is just a small part of CS. They do not have a monopoly on what is best in CS research, and not even TCS. In the last years, the lack of representation from other streams of CS has been noted, and only a few groups (mostly from Ivy League schools) are present there (to keep their prestige).
Given this situation, it is not surprising that some top researchers don't have many papers there. Trying to reinterpret the reality as you did above just shows how narrow is your vision.
I think we are getting something upside-down here: it is not good conferences that give "prestige" to papers by accepting them, it is good papers that give prestige to the conferences they appear in.
ReplyDeleteSTOC/FOCS did not become important conferences because a committee (or an anonymous online discussion) so decided, they became important because that's where most of the memorable theory work has appeared so far. Similarly SODA has become an important conference because good papers appear there every year. FOCS/STOC would become even more important if "mathy ML" papers and "logic in computer science" papers appeared there. And, as a fan of S/F, it would make me very happy if this were the case (especially for the "mathy ML").
So if a good paper is rejected from (or not submitted to) a conference, it is the conference that is harmed not the paper. The quality of the paper, and the likelyhood that it will become a seminal result, or the fact that it settles an important question, or the chances if will help its author get a job, are just the same if it is a technical report or if it appears in JACM.
So: (1) be proud of your results, not your numbers; (2) if you have great results, please send them to STOC/FOCS, and share some of your glory with my favorite conferences.
Here's another option: these are the people who are not getting published in FOCS/STOC.
ReplyDeleteThis is usually the last cop out of the "STOC/FOCS are perfect" crowd.
When presented with ample evidence that these two conferences, while undoubtly of high quality, are not the end-all and be all of TCS, they resort to taunts such as "sour grapes".
Very mature, and very rational.
An ode to FOCS/STOC...
ReplyDeleteI love myself
I want you to love me
When I'm feeling down
I want you above me
I search myself
I want you to find me
I forget myself
I want you to remind me
I don't want anybody else
When I think about you I touch myself
Just a note: Patrascu is not the only author of most of his publications.
ReplyDeleteNeither is Erik Demaine! What a couple of slackers.
The STOC/FOCS program committees consist of a representative set of respected researchers in theoretical computer science, and they do the best they can to select the most interesting papers among the submissions. The choices are not perfect, but I can't imagine how to do it better. If STOC/FOCS did not exist, I would focus on SODA, not have time to keep up with what's happening in the other conferences, and major breakthroughs in Complexity or other areas of TCS would happen without my being even vaguely aware of them. We are very lucky as a field to have this bi-yearly snapshot of recent research.
ReplyDeletePerhaps Patrascu can serve as an example of an extremely successful education, and as inspiration for ideas on how to educate the next generation. Again, we are lucky that such a talented young researcher has chosen TCS as his field.
The choices are not perfect, but I can't imagine how to do it better.
ReplyDeleteOh please. While I commend the work PCs do, it is not hard to come up with improvements:
First, increase the size of the PC to ensure broad coverage of TCS both in areas and in geographic regions. Of course stronger regions and areas would still have bigger representation.
Second, enlarge the definition of what is a good paper to include more than just "highly technical complexity/lower bound/trend-ish content" results. Looking back it seems that the top 25% papers submitted will get in regardless of their complexity/lower bound-ish content. It is the next batch of the papers where the conference seems to traditionally prefer technical difficulty/trendiness over other measures such as long term impact, relevance, applicability, etc.
Third, move FOCS back to parallel sessions. Having such a small acceptance rate magnifies errors while minimally increasing quality.
I'm sure there are more. Those are the ones at the top of my head.
"relevance, applicability, etc".
ReplyDeleteThat got a laugh out of me. Basically that's your way of trying to get trivial and useless results given the same importance as non-trivial ones. Good luck with that!
My God, imagine all the good work we could all be doing if we weren't spending so much time on this.
ReplyDeleteFOCS/STOC/SODA/ICALP are by no means perfect. But, if you get good results, and communicate them to the leaders of your field, they will be appreciated whether or not they have been published in the conference you would have liked. In particular, this will lead to getting good letters of recommendation, which are weighted as much or more by hiring committees than the number of papers you have in top conferences.
That got a laugh out of me. Basically that's your way of trying to get trivial and useless results given the same importance as non-trivial ones. Good luck with that!
ReplyDeleteI said applicable results, you said useless. I said relevant results, you said trivial. It seems reading is not your forte.
"Relevant" and "Applicable" are hopelessly vague terms on their own. Relevant to what? Applicable to what? You don't say. And in a field such as ours, one needs to be especially precise when using these terms...
ReplyDeleteIf you mean "making an appreciable difference in the real world", then I agree that work of that kind is very valuable. But I think such results are much rarer than technical advances. What is very common is to see people making vague claims (being precise is to their disadvantage) about the "relevance" of their work when said work has nothing to recommend it.
>I'm a quasi-loser for not having a PhD+tenure by this age.
ReplyDeleteThis is plain stupid. What about expanding your horizon and deflating your ego? A good point to start:
http://humanum.arts.cuhk.edu.hk/humftp/E-text/Russell/knowledg.htm
(Knowledge and Wisdom, Bertrand Russell)
Another issue about FOCS/STOC: It happened for me a lot that I had a very good paper a month or two after FOCS deadline. Then it appears one of co-authors is in the PC of next STOC. Then essentially you are forced to submit the paper to SODA, otherwise you need to wait for another year and even then there is no guarantee that the paper gets accepted to FOCS in the first try. This was a real issue for me and I think even for job markets, only counting F/S and ignoring SODA papers is not a good metric.
ReplyDeleteGood point about the PC conflicts. I now that at least two people who declined PC membership in SODA because they had several good results hatching. There is at least one oher person who didn't decline (and would have accepted even in hindsight), but still felt he had to apologize to all of his pending collaborators for shunting them that year from SODA.
ReplyDeleteThis is plain stupid.
ReplyDeleteGive him a break - he's young, PLUS, he's at MIT: what do you expect? MIT is about producing papers, not about gaining wisdom.
I close my eyes
ReplyDeleteAnd see you before me
Think I would die
If you were to ignore me
A fool could see
Just how much I adore you
I get down on my knees
I'd do anything for you
I don't want anybody else
When I think about you I touch myself
So much bitterness. This cannot be good for our community!
ReplyDeleteSome thoughts to cheer everyone up: I think that the "mafias" mentioned earlier may exist, but they don't form around selfish goals, but rather around research threads that they find thought-provoking. Instead of harboring bitterness and calling them names, I think that it is much more constructive to try to copy them and spread the excitement.
This is not a zero-sum game, and other researchers are not the enemy! I honestly believe that everyone in this community has the intellectual curiousity to fall in love with research ideas that truly are novel, insightful, and important.
ReplyDeleteThis is not a zero-sum game, and other researchers are not the enemy! I honestly believe that everyone in this community has the intellectual curiousity to fall in love with research ideas that truly are novel, insightful, and important.
The spirit of this is true.
But let's also face the fact that not everyone is equal and not everyone produces the same quality of work, and we have a lot of process and metrics to judge that.
In the end, some people don't finish their PhD. Others don't get jobs. Other don't get tenure, and so on.
While there are brilliant people in all of the above categories, they --- like the seminal results that got rejected from top conferences --- are the exception and not the rule.
Discussion is great, but people should be aware where they are coming from. That is, if they can.
About crypto. Nobody --- may be except the people working at MIT/Weizmann --- in cryptography counts the number of papers in FOCS/STOC. Those conferences are seen --- from the viewpoint of cryptography --- irrelevant. True, major crypto papers in 80s were published in FOCS/STOC, and sometimes a major paper is published there even nowadays.
ReplyDeleteThe reason is that only papers from a tiny fraction --- those of very theoretical kind --- of crypto get ever published in FOCS/STOC, while crypto is much much wider. It is applicable, and people care about practicality of constructions, exactness of reductions, they do cryptanalysis of existing schemes etc.
Instead of FOCS/STOC papers cryptographers count Eurocrypt/Crypto papers. But also this is controversial and not correct. Those people who work in symmetric crypto tend to instead publish at FSE, and in fact may never attend Eurocrypt/Crypto, considering most of the papers there too theoretical or just noninteresting.
The point: counting FOCS/STOC papers for a cryptographer is utterly irrelevant (it just shows you do a particular kind of cryptography). But of course counting any kind of papers is irrelevant. Do you know how many papers does Diffie have? But you know his name, no?
ReplyDeleteThe point: counting FOCS/STOC papers for a cryptographer is utterly irrelevant (it just shows you do a particular kind of cryptography).
Dude, we are talking about theoretical computer science, and by implication --- about the theory of cryptography.
Systems people also don't count STOC/FOCS publications. Neither do Chemists. Really.
Dude, we are talking about theoretical computer science, and by implication --- about the theory of cryptography.
ReplyDeleteI think you are the one who is missing the point. What Helgar meant is that theory of cryptography is much broader than what gets published in FOCS and STOC, so even theoretical cryptographers don't count FOCS/STOC papers.
Examples of theory of cryptography never published in FOCS:
ReplyDelete- construction of (say) short signature schemes without random oracles
- feasibility analysis of algebraic/differential/... attacks on block ciphers
- security analysis of triple encryption/modes of encryption
- ...
What is theory? If somebody comes up with a new factoring/disc log algorithm then it's a major theoretical result, but I've almost never seen cryptanalytic results in FOCS/STOC. (Except if it is quantum.)
Nobody --- may be except the people working at MIT/Weizmann --- in cryptography counts the number of papers in FOCS/STOC.
ReplyDeleteThis is a bit exaggerated, as many cryptographers are very interested in crypto results published in FOCS/STOC and actively try to publush there.
But even putting this aside, I have been told that tenure committees in US universities do count number of FOCS/STOC publications for cryptographers: since cryptographers are usually part of the "theory group", this is the metric that non-theorists know and trust (for better or, more likely, for worse).
PS: Using FSE as an example was a bad choice. As far as I am aware, FSE is almost universally considered weaker than Crypto/Eurocrypt.
Some time ago I found an article on a related topic, in the context of bio sciences.
ReplyDeleteAcademic Scientists at Work: Publishing at the Top of the Heap
We are not alone in the universe ;)
Piotr
I have been told that tenure committees in US universities do count number of FOCS/STOC publications for cryptographers:
ReplyDeleteSure they do. I do so too. That is not the issue. What we are arguing here is if they/we should look at FOCS/STOC only to the exclusion of everything else. This is what we have a quibble with. F/S are not representative enough of TCS to be used as sole measure. That is, "lots of F/S papers ==> theoretician is likely very good" is a reasonable assumption, while "few F/S papers ==> theoretician is likely not a superstar" is an unreasonable one. Too many good areas are missing in F/S, for whatever reason, for that argument to be made. A better judgment would be "too few F/S/SODA/Crypto-Eurocrypt/COLT/JACM/SOCG papers ==> theoretician is likely not a superstar", and even then, we can find counterexamples, but at least then we would be right the vast majority of the time.
"lots of F/S papers ==> theoretician is likely very good"
ReplyDeletetoo few F/S/SODA/Crypto-Eurocrypt/COLT/JACM/SOCG papers ==> theoretician is likely not a superstar
This is well-said. Let me add that the distinction between F/S and (for instance) SODA is that the implication
"lots of SODA papers ==> theoretician is likely very good"
does not seem to hold as strongly (hence the claims that SODA is not as strong on average as F/S).
Do you know how many papers does Diffie have? But you know his name, no?
ReplyDeleteDiffie is actually a terrible example. He had one extremely influential paper but did essentially nothing (at least in terms of research) since then. This is in contrast to, say, Rivest and Shamir who wrote (and continue to write) multiple important and influential papers after their famous 1978 paper.
In that case, it sounds like Diffie is an excellent example. If he earned almost as much renown as RSA by writing only one influential paper, what does that suggest about the strategy of writing as many papers as possible?
ReplyDeleteIn that case, it sounds like Diffie is an excellent example. If he earned almost as much renown as RSA by writing only one influential paper, what does that suggest about the strategy of writing as many papers as possible?
ReplyDeleteIt's better to be lucky than good?
Since the point of theoretical computer science is solely to recognize who is the most badass theoretical computer scientist, I can only say:
ReplyDeleteGO HOME PUNKS!
WIGDERSON OWNS YOU!
Why oh why won't funding agencies give out more grants for the purpose of writing more and more STOC/FOCS papers so we can once and for all settle who is the ultimate master of TCS champions?
ReplyDeleteA reply to Tom Hagen:
ReplyDeleteWell, I guess you are a nice person that have faith in human nature in general. However, I must emphasize that humans do have a selfish side, and it will come out if there's nothing to put it under control. And note that the "mafias" do not only exist in the theory field, they are everywhere. But, is it really just about the topic of the papers? Consider the following story: a friend of mine (the one that talks about the "mafias") once submitted a paper, got a strong reject with terrible reviews. The major reason of the reject is "insufficient experiments". A while later, he met some professor in a conference, who basically tell my friend to "back off, don't touch our research topic"..... and then still a little while later, he sees a similar idea being published in an even bigger conference (well, he believes there's no copying here). How much experiment does this published work have? Even less than the version submitted by my friend.....
A general comments to many who have posted here:
There is no point of arguing whether we should evaluate the quality of a researcher (especially a PhD student) by the number of publications he/she has at one or two top conferences. Why? Because people who are responsible for evaluating them and giving them jobs/funding love to simplify things like that. It is so convenient for them to reduce the fate of everybody into one or two numbers.
And, what's wrong with reducing things into one or two numbers? It's perfectly fine if those one or two numbers really have meaning. HOWEVER, it appears really not the case. As I have mentioned, the flaws in the process is making those one or two numbers very favorable to some (I call them the mafias), and almost random for others (they are the small potatoes, like me).
Once again, you can call me sour grape, BS, whatever. But I assure you, you'll hear it over and over, either from me, or from somebody else. I now offer you a chance to shut us (or at least, me) up once and for all: implement the rejected papers collection system I've proposed. If no strong evidence of the existence of nonsense reviewers and the so called "mafias" is found in two years, you can call me whatever you want. And you will hear no more about the mafias from Thomas Moore.
I can't believe I'm adding a comment to this post, but just to make sure new students don't get the wrong impression here are some clarifications:
ReplyDelete1. our job is one of the most creative, and can not be reduced to any number.
2. in hiring and promotion people are well aware of that, which is why we use *reference letters* and not counting when making such decisions.
3. stoc/focs (or nay other conference journal) are the equivalent of a newspaper, where the commmitee informs the audience of interesting developments in the field. Like Luca said, if a conference missed a development then it's the conference that hurts, not the researcher.
4. Just as you wouldn't read only one newspaper, you shouldn't follow only one conference, regardless of what your field is. In my areas of theory of crypto and derandomization I can think of several breakthrough papers that did not appear in STOC/FOCS (examples that come to mind are the cramer-shoup cryptosystem, Boneh Franklin identity-based encryption, and Impagliazzo-Kabanetz-Wigderson paper providing the first indication that uniform derandomization implies lower bounds)
Sometimes conferences miss out on a great result because of a mistake by an overworked committee, often the authors simply do not submit their paper. Of course as an author you should submit to whatever conference you find more appropriate, but please do not refrain from submitting to FOCS/STOC because of some uninformed anonymous weblog comment.
--Boaz
Firstly, thanks to Boaz for the post. Like him, I was torn between leaving a "Mafia" post as a "conclusion" of the thread, or replying to a thread that contained, at times, pretty offensive stuff.
ReplyDeleteBut given the new beginning, there are some things that I would like to add. Basically, I think there is some middle ground between many of the opinions expressed here (the "middle" is not exactly in the middle - I mostly agree with Boaz, and mostly disagree with the opinions he addressed).
Now to bussiness:
3) "If a conference missed a development, it is the conference that hurts, not the researchers".
Well, I think it is fair to say it hurts both.
The conference becomes less interesting and potentially less relevant. If this happens too often, people (submitters and attendees) move elsewhere.
At the same time, the researcher does not get the benefit of broader publicity that comes with publishing at a top conference. As a result, paper can have less impact than it might have had otherwise. And, like it or not, acceptance to a prestigious conference is a form of quality stamp (approximate, probabilistic, but nevertheless).
Like Boaz said, the conference names are, by far, not the only factor in evaluating a researcher - people ask for reference letters etc. That said, people ask for a CV with a list of publications and venues as well. The more different sources of information, the better.
After all, all the PCs I've been to tried to select "high quality" papers and reject the "lower quality" ones.
Which brings up another important point that I think is relevant to the discussion: the "quality" of a paper can be, very much, in the eye of the beholder. Like Boaz said, the quality of research cannot be reduced to a number. There are many aspects of a paper: closing an important open problem vs opening a new direction, theoretical insight vs practical impact, etc etc. There was a looong discussion on this blog devoted to this.
People attach different weights to these criteria. They can also have different perceptions of a contribution for each of them (after all, it is often hard to evaluate a new direction before the paper is published, when it really did not have a chance to impact anything yet). Add to this some random mistakes and, unfortunately, an occasional self-serving bias left unchecked, and you can easily have different rankings by different people. Which does not exclude a significant level of agreement (which is often the case), but does not exclude a disagreement either.
So: there are and there will be disagreements about rejections of papers. This is because contributions of a researcher are harder to evaluate than, say, of a 100 meter runner.
Instead of a conclusion, here is a nice article on rejection. Different area, but nevertheless quite relevant, I think.
Academic Scientists at Work: I Can't Believe They Didn't Like It!
Piotr
If you guys believe that those in power are benevolent, why do you need your constitution? Why you need free press? And is anything different when it comes to academia? Are those that have say on things, have power incorruptible and always correct when it comes to academia? Why would you ignore or stone one who propose a monitoring mechanism?
ReplyDeleteBoaz, thanks for restoring some civility to the discourse. But I believe your comments are more misleading than any of the previous ones. Certainly, in an ideal world, all the things you said would be true. It is clear from all the invective in this comment thread that people believe the situation is far from ideal. I think it serves the community better to acknowledge this, and perhaps to discuss ways of changing it, than to pretend everything is fine...
ReplyDeletePiotr has already addressed the ridiculously naive contention that a conference derives all its importance from the papers presented there. This should be true, but isn't in practice, and never will be, for the simple reason that people are busy and it's much easier to judge a paper by whether it gets into a conference than to personally assess its "intrinsic value".
You also talk about hiring and promotion people as if they are typically objective, dispassionate and deeply concerned about judging the "creative ability" of researchers using a combination of factors. This must certainly be news to people who have actually served on such committees. It's no secret that factors like the trendiness of a research area and the reputation of the researcher's advisor (completely independent of the researcher's ability) get far more weightage than they should. And reference letters often increase the impact of these factors. Counting conference papers is certainly flawed, but it is at least objective.
To sum up, a lot of the frustration in evidence here has ample cause. New students should be prepared for this, but on the positive side, the creative satisfaction from doing research can make it all worthwhile.
What is interesting (to an outsider) is that TCS and CS in general has this "conference publication is everything" culture. In most every other academic field (from various kinds of engineering to performance studies) conferences are expensive to attend and the real results appear in journals. One exception is the Modern Language Association (MLA) conference to which grad students go for job interviews.
ReplyDeletePersonally, I think a lot of this FOCS/STOC vs. SODA "who has the biggest cock" fight would get sidelined if journal publication became the norm rather than the exception in CS. Or am I missing something huge?
Personally, I think a lot of this FOCS/STOC vs. SODA "who has the biggest cock" fight would get sidelined if journal publication became the norm rather than the exception in CS.
ReplyDeleteI don't see how this would change anything. The same problems would still remain (even worse, since editorial boards change at a glacial pace compared to PCs), and then the fight would come down to whether SICOMP or JCSS have the bigger, er, brain.
Since we're talking about controversial issues:
ReplyDelete1. Why is David Johnson a permanent member of the SODA steering committee?
2. Are there any other permanent members of streering committee of other conferences?
3. Why don't we change to an anonymous submission system? The current system has a problem with people helping each other to get papers in. As everybody can see from people's opinions, the research today is about having papers in certain conferences and not about actually advancing our knowledge.
And BTW, in case you actually think this is not the case, take a few researchers with lots of publications, and look at the distribution of where the papers (by year per conference) got accepted. You might notice a big variance - like people getting 5+ papers in one year, and none the next. Do you think it's natural? (like they were incredibly smart and productive a year and not the next?) Giving the time limitations I would say no, unless there's a bias with the program committee. If you look closer you might even find a connection. Therefore, we do need an anonymous submission system.
You might notice a big variance - like people getting 5+ papers in one year, and none the next. Do you think it's natural? (like they were incredibly smart and productive a year and not the next?)
ReplyDeleteThis is plain tin-foil hat stuff, and likeky posted by someone who has never served in a PC.
While PCs are not free from bias (we are, after all, human), any such bias is a lot more subtle than that: all papers are refereed by three PC members, close friends usually self-disqualify due to conflict of interest and reviews are made public to other PC members.
If someone were to give an inflated review to a friend other PC members would notice and looks bad, so there is a great incentive not to do so. Still, occasionally it does happen and one get to see a review that stands out as unbiasedly too positive or negative as compared to others. In those cases the PC chair usually asks for yet another review from a fourth party that settles the issue.
According to citeseer impact factors
ReplyDeletehttp://citeseer.ist.psu.edu/impact.html,
the impact factor of FOCS and SODA is equal to 1.51 and the impact factor of STOC is a bit higher (STOC: 1.69)
So it seems stupid to think that SODA is much different than FOCS/STOC in terms of impact factor in theory. Yes, FOCS/STOC have some crazy math papers probably wihtout any applications, but doesn't say anything about the impact factor in theory. Note that SODA has almost 450 submissions now and every year this number increases by 10-15% and SODA becomes more competitive, though STOC/FOCS didn't have changes in the number of submissions.
Somebody sent me the link to this discussion. Since this page seems to be getting attention from many other sites, I should correct several misimpressions created by the various posters. I have served on several PCs in the past and most recently as PC chair of FOCS'06. Please also read Piotr and Boaz and Luca's posts thoughtful comments.
ReplyDeleteThe last anonymous poster gave you some insight into the PC process, about how your decisions and ratings are scrutinized by 20+ other PC members all of whom are prominent researchers (some whom you don't know), and how at the physical PC meeting you are expected to explain your rating to them. It is hard if not impossible for PC members to get their friends' papers in if the papers can't stand up to this scrutiny (not to mention that there is a conflict of interest policy).
Sure, there is a tendency to go for trendy topics as in any other discipline. But honestly, my impression over the years is that people are usually tougher on papers in their own areas. To give an example, the recent FOCS PC had many experts in economics, games, etc. yet we seem to have a lower-than average number of papers in this trendy area. My guess is that if we had had fewer experts on economics and games, then we would have been *more* vulnerable to err on the side of trendiness.
However, I do readily agree that
the decision on marginal papers is
heavily influenced by the makeup of the PC (as it would be in any decision process, whether journal or conference). These are probabilistic, noisy events after all.
Another comment: people who have never served on PCs ---I was such a person not so long ago-- may have the impression that this is a position of great power that people look forward to. ("Aha; now is my chance to get in the papers of all my friends!") In reality, people who do this more than once think of it as a "service to the community." Realize that the PC is asked to review 250-odd papers (as many as several years' intake at respectable journals) in a period of 1.5 months or so. This commitment usually comes in the middle of an academic semester, and has to be met within the constraints of a busy professional life. Hardly something most people would volunteer for. (That said, being on the PC does reward you with a broad overview of goings-on in the entire field, and this is very nice.)
Ditto for positions such as steering communities and so forth. Usually the problem is not a surfeit of qualified candidates for such positions ---rather the opposite. Qualified people have to be talked/charmed/guilt-tripped into doing such community service.
Final comment: it should be realized that the composition of the PC is also a highly random variable. When chosing my PC 7 months ago I did not think of any of the factors mentioned earlier apart from balance of research areas. The budget for the PC meeting was about $400/member, so I focused on finding members who could reasonably drive to Princeton, NJ. I'm guessing several PC members who flew will pay part of the expenses out of their pockets. Many had to say No for various personal reasons (birth of a baby, etc.). I had a period of maybe a month or two to find 20+ people. Many say no, sometimes after a few weeks indecision. So you scramble to find somebody else (preferably geographically close) to cover the same area. etc. etc.
Very noisy process; again this would reduce the effectiveness of any alleged "cliques."
I hope all this information was useful to you.
Sanjeev Arora
Princeton University
Why does the PC chair choose the committee? This puts a huge responsibility on one person to ensure that different areas in theory are adequately represented. It is not hard to imagine different, more distributed ways to choose a committee...
ReplyDeleteIf you have been in research long enough, I suppose you should have heard of, or even have personal encounters with the "Mafias": powerful schools/individuals who have some say in PCs of some "top" conferences, and who are seen to have underserved advantage in this peer review process.
ReplyDeleteIs there a STOC/FOCS program chair/committee member that does not agree with the previous sentence ?
"lots of SODA papers ==> theoretician is likely very good"
ReplyDeletedoes not seem to hold as strongly
Sure enough.
(hence the claims that SODA is not as strong on average as F/S).
This does not necessarily follow. Let me give you an example:
Two construction workers set out to compete for the title of best builder in theorytown in side-by-side construction sites. At the end of the day one buider has put up the four load bearing walls of a house, while the other builder put up those four walls plus another twelve.
Yet, the first builder claims he's a stronger better builder because all of his walls are load bearing, while only 1/4 of the other builder's are. :-)
I don't know what to conclude from this parable, except that load bearing walls are better than the other kind.
ReplyDeleteIf you have been in research long enough, I suppose you should have heard of, or even have personal encounters with the "Mafias": powerful schools/individuals who have some say in PCs of some "top" conferences, and who are seen to have underserved advantage in this peer review process.
Is there a STOC/FOCS program chair/committee member that does not agree with the previous sentence ?
The truth is you don't have to be a mafia to exert underserved control over a PC meeting, you just have to be loud and/or argumentative and/or forceful. Most committee members feel pretty passively about at least 70% of submitted papers, and thus are willing to cave into strong pressure of any sort.
I think an important question is whether people make the right choices about what to work on.
ReplyDeleteExample:
Suppose at the beginning of the year I tell you that you can have one additional STOC/FOCS paper OR a 10% chance of proving that RL=L.
As a field, we should certainly hope that people would make the second choice, because a proof of RL=L is easily worth 10 standard F/S papers, but I am pretty certain that many people would choose the first option over and over again.
You know, this whole discussion reminds me of how some undergraduates whine
ReplyDeleteabout grades. ``Can the entire learning process really be condensed into a number?''
they cry. Well, guess what? The answer is a big yes, in that there
is excellent correlation between scoring high in a course, and understanding
and being able to apply the material of the course (which are the goals
for an undergrad course).
Which is why the comments above, sugarcoating the truth, are so disappointing.
A major part of the evaluation system for scientists is by the quality and number of
their publications, with the publication venue being a good indicator for
quality.
Yeah, a conference rejecting what turns out to be fundamental progress is bad
for the conference, in the same way that a smart student getting a poor grade
might be a failure of the exam process. But is that really the common case?
No, in the long run there is excellent correlation between a paper
introducing great results and being published in a top conference.
People should just learn to evaluate themselves fairly. Constantly failing to publish
at top conferences is just not a good sign. It should cause one
to take a hard and fair look at themselves and perhaps adjust their goals, work
harder or move to another field --- not to invent conspiracy
theories about why their work is misjudged or to belittle those who are more
successful.
That being said, it's ridiculous that all the ``really, don't worry about the numbers''
comments are from people with 40% or more of their conference publications in
STOC/FOCS. Are the ivory towers really that high?
I don't know what to conclude from this parable, except that load bearing walls are better than the other kind.
ReplyDeleteThat a good builder is the one that builds all the walls that are necessary (and no more), not just the load bearing walls.
In terms of conferences it means that a good non-specialized conference accepts all the submitted papers that to their best of their judgement are worth telling other non-specialists about, and leave the rest for the specialized conferences (SPAA, CCC, etc).
If a conference for the sake of low acceptance ratio alone rejects a paper that was worthy of publishing at such venue this reflects bad on the conference not the paper, as Luca pointed out.
p.s. However, I agree with Piotr that the consequences of such mistakes are borne by both the conference and the author.
"You know, this whole discussion reminds me of how some undergraduates whine
ReplyDeleteabout grades."
1. You're begging the question altogether, since you start from the premise that FOCS/STOC is perfect. The problem is not a general one about "being evaluated by numbers", but rather specific, about the failure of certain "top" conferences to serve as the right yard-stick.
2. The biggest problem is not that excellent papers are rejected by "mafias", rather that mediocre/trendy papers are ACCEPTED by "mafias".
The knives are sharp,
ReplyDeletebecause the stakes are small.
Honestly, if you have such a problem with STOC/FOCS, why not publish elsewhere?
I give up. Looks like my post had no effect. Where exactly does anybody see the possibility of mafia takeover of the PC in the process I and others have described?
ReplyDeleteDo not-so-good papers get accepted to STOC/FOCS? Sure, it is impossible for the community to write 60-70 amazing papers every six months. However, the following is the fallacy in some of the above assertions: "Some marginal paper got in ---> my paper should have got in."
It is a fallacy because the vast majority of papers are at the margin, and not all can get in.
Sanjeev Arora
"It is a fallacy because the vast majority of papers are at the margin, and not all can get in."
ReplyDeleteThis is exactly the point that I think SODA is much better than F/S (because it is not that concentrated as F/S about PC). In SODA, usually the marginal papers which are strong enough will get accepted and there is not a real issue about only some marginal papers that will get accepted in F/S because of ''mafia'' or better to say some PC members who really like their friends or at least their areas of research. This is especially important since some people want to count these marginal papers accepted in F/S, ignore other nice works, and then make lots of deal out of it for job/tenure/...I really do not think that anyone has any issue with counting 25% top papers in F/S.
Sanjeev,
ReplyDeleteDo not despair. The vast majority of readers are of the silent kind, and most of us appreciate very much that you took the time to talk about the latest FOCS round. Moreover we do not share the radical positions held by either side: F/S is perfect vs. F/S are mafia run. If one concentrates on the thoughtful posts (Luca, Boaz, Piotr, you and certain anonymous messages) one can tease out a clearer picture of things.
No-one has responded to my question of why the committee chair is responsible for picking the entire committee. This makes it easier for people to level charges of bias. I believe that such charges are to a great extent unfounded, but it does seem strange to adopt such a unilateral system.
ReplyDeleteBy the way, who is the PC Chair of STOC'07?
ReplyDeleteSo what do you think about somebody who publishes 30%+ of papers is STOC/FOCS/SODA? He's a guy with good intentions, but not God...
ReplyDeleteThis is actually true; God doesn't have any STOC papers.
PC chair for STOC'07 is Uri Feige.
ReplyDeleteBTW, I agree that the community should be open to experimentation with new systems and ways of doing things. For instance, double-blind submissions seems like a reasonable idea.
The appropriate forum for raising these ideas are the business meetings at STOC/FOCS. Contact the SIGACT/IEEE chair before the conference for permission to make a motion from the floor. Anticipate all obvious objections and prepare good answers to them.
Having a larger support structure around the PC chair is also a good idea, since currently he/she has too many responsibilities.
Again, the proper description of the situation is not "one person assuming too much power for six months" but "too many responsibilities dumped on one person for a six month period" (which may lead to suboptimum performance).
Sanjeev
Having said that Mihai and Assaf have three papers in this FOCS and it is the maximum, it would be fun to have the records of maximum papers per person of top conferences over all years (esp. if you follow FIFA'06, they have also lots of records:) ). I think maximum for FOCS and STOC is 4, maximum for SODA is 6 and maximum for SOCG is 8. Is there any more information in this regard about other top conferences or even confirming the above information?
ReplyDelete---- In SODA, usually the marginal papers which are strong enough will get accepted and there is not a real issue about only some marginal papers that will get accepted in F/S ...
ReplyDeleteIn SODA'05 business meeting, Adam Buchsbaum showed a slide with a distribution of paper scores (slide 14). An apparent conclusion is that here too the "margin" is pretty thick, and given the upper bound of 135 papers (out of 491), many or most papers in the margin are rejected. See Geomblog's SODA business meeting redux for more on this topic.
Oops, actually it was 136 out of 487 (4 papers withdrawn, 2 papers merged into 1).
ReplyDeletePiotr
Since everybody's obsessed with numbers to some extent, why not make some statistics of the papers from STOC/FOCS, SODA, ICALP, CRYPTO, etc, based on authors, field, university, country, faculty/postdoc/grad/undergrad etc.
ReplyDeleteIt'd be interesting to see who owns who. :))
MIT certainly has lots of papers this coming FOCS.
The information is all there.
Hi, I am a first year PhD student working in theory. I am surprised to know that FOCS/STOC are not double-blind. Many top conferences in other firleds of CS are.
ReplyDeleteWell, people seem to be arguing that at least the program committee is blind.
ReplyDeleteReplying to Mr Arora concerning "where exactly does anybody see the possibility of mafia takeover of the PC process":
ReplyDeleteI believe the root cause of such unfortunate "misunderstanding" is because the PC process is basically a BLACKBOX process.
More often than not, those who submit a paper to the "top" conferences have not, and will never serve in a PC of such conferences. As a result, all the
information they get are from the following origin:
1) personal experiences, the review they (or their friends) get,
2) the accepted papers
3) the description of this process by some kind insider
Obviously, this is very far from perfect information, and is heavily influenced by the authors' personal experience. And of
course, they start to develop what you may call "sour grape" syndrome when their papers got rejected, by some totally BS
reviews. To make things worse, when they look at the accepted papers, they find plenty of work with the same, or even lower
quality than theirs. Furthermore, to make the grape even more sour, these work are mostly (though, not all) from big names/schools in the field, and, CAN VERY WELL BE REJECTED BY THE SAME REASONS THAT REJECTED THEIR PAPERS AT THE FIRST
PLACE!!! At this point, the grape grows so sour that one simply cannot use the information revealed about the PC process to
refute the already formed theory: there are mafias.
And, to have a little clarification, the mafias do not have to "takeover" the entire PC of 30 or more people. Why? Consider
what others have posted here previously:
1) most committee members feel pretty passive about at least 70% of submitted papers
2) there are plenty of marginal papers
So, here is a little imagination of Thomas Moore (who has never served in a PC of conferences like STOC/FOCS), about how an extremely small number of mafia members (one, or maybe two) can put their mafia friends in undeserved advantage, and kick out almost all non-mafia submission (except those really good ones):
1) If the mafias have interest in the fate of a paper, it usually means it's related to their work => they are the expert of
that area, and they'll most likely review that paper (I believe this is especially true in "top" general conferences. As I have argued before, experts from any particular area are of limited supply, and external reviewers are not always called in. See Lance's blog about low quality comments from "top" conference and you'll see how limited supply these experts are).
2) consider the paper in this example one of the many marginal papers, and receive either a weak accept/reject from the
other reviewers
3) imagine that, the reviewers are the only one who read the paper in the PC, and the non-mafia reviewers spend less than 1
hr on the paper (how many papers they are reviewing again???).
What do all these sum up? If the one mafia reviewer express strong opinions (either positive or negative) about a paper,
he/she can significantly influence the fate of the paper. Since it's marginal, a little bit more positive may get it in, a
little more negative can kick it out. Since the mafia reviewer has an objective in mind, of course he/she comes prepared to explain his/her reviews. And please agree with me, if it's a marginal paper, one can easily dwell on some of its
strength/weakness, and magnify it enough to form compelling reasons. The remaining non-mafia reviewers will not argue with the mafia reviewer, since they are quite neutral about the paper, and in fact, don't really know it that well (afterall, they spend less than an hr on it). The remaining
of the PC haven't even read the paper (remember, most of them are not expert in the specific area, since the PC of a "top" general conference is usually made up of people with greatly varying expertise), they will rely on what the reviewers tell them, especially the mafia one (who's the loudest).
Of course, this is all the imagination of Thomas Moore, a serious sour grape syndrome sufferer. But, can we blame Mr Moore for such imagination? Afterall, there seems very little hard fact he can use to explain the things he sees, about what got
published and what got rejected, except, of course, the insider information from people like Mr Arora. But should we live the BLACKBOX process in our "top" conferences? Should we evaluate these conferences based on, mostly, faith? If your answer is "no", speak up.
The things that Mr. Moore says are really not that far from reality, both from my past experiences in F/S PCs and also from those that I heard from other PC members.
ReplyDeleteCan we all at least agree that the following would be desirable for STOC and FOCS -
ReplyDelete(1) Larger and more representative PCs
(2) More reviews per paper
(3) Better reviews - if it's clear from a review that the reviewer hasn't spent enough time on the paper, get another one. Even if at the cost of lengthening the decision process by a couple of weeks...
And none of the above are particularly costly or difficult to implement, are they?
making STOC/FOCS double-blind is definitely a first step in building some kind of confidence in the community.
ReplyDeleteAnd of course, they start to develop what you may call "sour grape" syndrome when their papers got rejected, by some totally BS
ReplyDeletereviews.
This only hurts the field. While serving on PCs I've often seen rather weak papers who are unanimously given low marks but the comments to the authors are "fix this typo, improve the conclusions section".
One can why authors might be upset with the rejection. Can we not give a better list of reasons than this? If one problem is time, one could even envision a system that would provide better feedback while saving time. Say, imagine an HTML form (if SIGACT ever becomes web aware) with the following check boxes:
MARK ALL THAT APPLY:
[ ] Accept
[ ] Implement corrections, then we suggest resubmitting to [_______].
[ ] Result not quite up to the rarified standards of X, submit instead to [_______].
[ ] Results seem interesting, presentation is horrible. Rewrite.
[ ] Have native English speaker correct numerous errors.
[ ] Bibliography is poor, which makes reviewer suspicious as to novelty. Review previews work, starting from [__________].
[ ] Reviewer has questions about the validity of Theorems [______] rewrite explanation more carefully.
[ ] Result is too incremental. Authors should consider doing [________].
and so on and so forth.
making STOC/FOCS double-blind is definitely a first step in building some kind of confidence in the community.
ReplyDeletePersonally, I'm not sure if it would make a difference, but then again when I see how vehemently opposed some big names are to it (and mostly for rather bogus reasons**), it makes me wonder if these people are not subconciously responding to the loss of a system that does indeed benefit them.
** among the bogus reasons is that one knows who wrote the paper anyhow. In practice, in other fields where this happens, this has proven not be the case. Plus if by accident the reviewer does happen to know with near certainty who wrote the paper, then it is incumbent upon that person to recuse him/herself.
there are two ways to crib
ReplyDelete1) why did my paper get rejected
2) how did "the other" paper get in
the HTML check boxes might solve (1).
Regarding (2).... the reviewers can sit and write a 5-10 sentences about "how this paper deserves a place in F/S", quoting some applicability, relevance, importance of the results. This is not too much to ask for, since the reviewers already have their reviews.
It would be great, if this review (and may be the abstract) is uploaded next to each accepted paper in the accepted list.
I understand that PC members are busy, but I think that there is no reason that the sub-referees that do the actual expert reviewing could spend 4-8 hours on each paper and write-up a nice 1-2 page report intended to help the submitters out. The effort wouldn't kill the reviewer. I do so regularly and it is just fine.
ReplyDeleteOf course, there is little incentive for this sort of thing, especially for pretenure people.
I agree, nothing seems more natural than blind review. What exactly are the arguments against it? "We would know who wrote the paper anyhow" doesn't seem like a strong argument -- even if this is so in some cases, its no worse than the current system, and surely in many cases, it will be better. Are there stronger arguments blind review? Double blind experiments are standard practice to eliminate bias...
ReplyDeleteAs Sanjeev said, anyone with such suggestions is welcome to raise them at the STOC/FOCS business meeting. Just note you should prepare a more serious proposal than some half-baked weblog comment, including data from the experience of other similar conference (for example CRYPTO used to have non-anonymous submissions, then anonymous submissions, and CRYPTO 06 went back to non-anynomous).
ReplyDeleteIt's important to provide as much meaningful feedback to the authors as possible, but keep in mind that the PC is working under a tight deadline and their main goal is to select a program and not to provide a detailed review.
From my experience on PC's, people are generally very "un-mafia" - if anything, like Sanjeev said, experts are often harder on papers from their own subfield. In any case, this hypothetical "mafia scenario" is much more likely to happen at a journal than at a conference, where you have a completely new PC every 6 months.
While it's definitely idiotic to treat STOC/FOCS acceptance as a perfect (or even imperfect) measure of basically anything, you should realize the volunteers running them do a pretty hard job. No matter how transparent this process will be, there will always be complaints. Some of the suggestions on this post (publishing titles and reviews for rejected and accepted papers) also violate the authors' privacy.
Bringing it up at a STOC/FOCS business meeting is all well and good, but there is nothing wrong with discussing the merits on a blog. Several posters have argued for double blind review, and I haven't heard any arguments for non-blind review, nor can I think of any good ones. Obviously this has been debated before if CRYPTO switched from non-blind to blind, and back. What were the issues debated there? I would find this enlightening.
ReplyDeleteThanks!
....but keep in mind that the PC is working under a tight deadline and their main goal is to select a program and not to provide a detailed review.
ReplyDeleteI agree that the volunteers of F/S do a pretty hard job and I appreciate it.
Most of us work for atleast one year (or 6 six months if u r extraordinary) for a F/S paper. It is frustrating to see papers getting rejected with tiny reviews.
As Piotr pointed out earlier, this might hurt both the conference and the authors.
(for example CRYPTO used to have non-anonymous submissions, then anonymous submissions, and CRYPTO 06 went back to non-anynomous).
ReplyDeleteThis paragraph is rather misleading. It makes it sound like Crypto tried anonymous submissions for a short while. Truth is Crypto went to anonymous reviews in 1989 and did fine under this system until this year.
For a bogus argument against anonymous reviews see Oded's
opinion on the subject: "it sugests that program committee members cannot be trusted to make a decision based on merit". This same argument was bandied about when medical experiments moved to double blind protocols: "are you suggesting that eminent doctors are cheaters that cannot be trusted?" went the standard argument.
Of course not, what people in medicine and in TCS (with the possible expection of Thomas Moore) are suggesting is that it might be hard to keep subconcious biases from creeping in.
I am surprised that nobody has mentioned this so far, but there is an obvious difference between journals and conferences: In journals (that i know of, at least) you can *reply* to the reviewers comments and in most cases revise, and get reviews again, reply again etc.
ReplyDeleteThis ensures much better quality since you can defend your work, and also makes the reviewers more carefull.
Of course, there is a tradeoff: Better quality in reviewing=More work for the committee and delay for the publication.(some journals take years to publish, but what is the rush in CS theory anyway? its not like systems). If you allow a revision and second round of reviewing, you have much less noise, but at some cost.
So here is one simple idea that does not add too much complexity, but i think will greatly reduce noise:
Before the decision, send the reviews to the authors. Give them two days to reply. PC takes reviewers comments AND reply into account to decide. (I think this will also make the reviewers more reliable. )
In fact, in the eurocrypt 06 business meeting, there is a vote count on non-anonymous submssions, over 99% people are against it.
ReplyDeleteSee:
http://helger.livejournal.com/31506.html
I prefer submitting to journals, because of the to and fro review process. Of course this does not apply to those who want to publish at a faster rate and make themselves happy with the "numbers".
ReplyDeleteI also don't pay attention to "conference is better than journal" myth.
btw, does someone know about the origins of this myth in TCS.
Most people want to publish at a faster rate not because they want to make themselves happy with the "numbers", but because they have to survive in this field. Not enough papers in 3 (or 6) years mean no tenture (or no job for a grad student).
ReplyDeleteHaving "numbers" make those tenture/hiring decision making people happy.
... and CRYPTO 06 went back to non-anynomous
ReplyDeleteEven further, Helger's blog seems to imply that Crypto 06 being non-anonymous is a one of: a unilateral experiment of the PC chair. Can anyone comment on this?
I also don't pay attention to "conference is better than journal" myth. btw, does someone know about the origins of this myth in TCS.
ReplyDeleteReality?
I've had three papers rejected from a journal in my lifetime. I can't even remember how many papers I've gotten bounced from conferences ...certainly over several dozens. I know this to be the case for most other people I've discussed this with.
What does everyone think about the usefulness of PC meetings? It seems that having good review software makes it possible to reach decisions without an actual meeting; also, since the discussion in such cases is done through the system anyway, it's plausible that the authors may end up with more meaningful comments. And it totally eliminates the nightmare mafia scenario with one PC member dominating the discussion at the meeting and swaying the rest of the committee. Not to mention the costs of having a meeting!
ReplyDeleteAnonymous:
ReplyDeleteI've had three papers rejected from a journal in my lifetime. I can't even remember how many papers I've gotten bounced from conferences ...certainly over several dozens. I know this to be the case for most other people I've discussed this with.
Many people only send an article to a journal after the some form of the article has appeared in a conference. If you do the same, then the events you describe are consistent with journals and conferences accepting similar work, with journals giving out more thorough reviews and catching more mistakes.
Lord Fizzlebottom
So here is one simple idea that does not add too much complexity, but i think will greatly reduce noise:
ReplyDeleteBefore the decision, send the reviews to the authors. Give them two days to reply. PC takes reviewers comments AND reply into account to decide. (I think this will also make the reviewers more reliable. )
This year, SIGMOD implemented a similar idea. My take on it was that (a) indeed it did help reduce the noise, but (b) it increased the complexity of the process as well. In more detail:
(a) If there are any "not-fully-justified" complaints about the paper, the authors have typically the best knowledge (not to mention motivation) to provide a detailed rebuttal.
(b) Drawbacks:
- the preliminary reviews must be done earlier (or the whole process takes longer), since the authors need time to respond, and so do the reviewers, including the external ones
- the authors must make sure to be on-line during the feedback process.
Piotr
This year, SIGMOD implemented a similar idea.
ReplyDeleteSo did AAAI this year. We'll see what people have to say during the business meeting.
"What does everyone think about the usefulness of PC meetings? It seems that having good review software makes it possible to reach decisions without an actual meeting; also, since the discussion in such cases is done through the system anyway, it's plausible that the authors may end up with more meaningful comments. And it totally eliminates the nightmare mafia scenario with one PC member dominating the discussion at the meeting and swaying the rest of the committee. Not to mention the costs of having a meeting!"
ReplyDeleteThis is one of the reason that SODA has a better situation in this regard, since there is no such a centeral meeting in which one PC member can do everything.
Helger's blog seems to imply that Crypto 06 being non-anonymous is a one of: a unilateral experiment of the PC chair. Can anyone comment on this?
ReplyDeleteWhat does this even mean? My understanding was that the PC chair simply prefered non-anonymous to anonymous; since there was no official policy either way, why shouldn't she get to choose?
For what it's worth, the two arguments I have heard in favor of non-anonymous submissions are: (1) in case of doubts of correctness, relation to previous work, etc., knowing the author may be helpful (if a well-known researcher with 20 publications on derandomization says a result is new, you can believe it; if a 1st year graduate student says the same you might question it); (2) it can help eliminate the so-called "mafias" by putting potential conflicts of interest in the open.
I am not saying I agree with either of these arguments. Personally, I prefer to judge a paper completely on its merits, and have sometimes found that when I know the author of a paper it makes it that much harder for me to be perfectly objective as a reviewer.
--Jonathan Katz (member of Crypto 06 PC)
What does everyone think about the usefulness of PC meetings?
ReplyDeletePeople have studied face-to-face meeting dynamics vs electronic gatherings. What they have found is that compromise is nearly impossible to achieve electronically. The flip side of this is that a vocal person will bully his/her way through. Also it has been observed that if someone proposes a harebrained idea, using email it would be promptly shut down, while in face-to-face meetings people are less keen to express just how inane they believe the proposal to be and will even agree to implement part of it, just to let the proposer save face.
Are PCs a case of meetings that benefit from face-to-face compromise or more the case of a meeting where the bully dominates?
Certainly the PCs with electronic meetings that I've been part of had never come to an impasse. Invariably after a few rounds of back and forth a consensus seems to appear. If it doesn't chair asks for new impartial opinion and majority carries the day.
Me: Crypto 06 being non-anonymous is a unilateral experiment of the PC chair.
ReplyDeleteYou: What does this even mean?
It means that it was a decision taken by a single person on her own and not an agreement of the community moving away from anonymous submission, the community having reached the conclusion that anonymity was a failed experiment, as the original poster seemed to imply.
since there was no official policy either way, why shouldn't she get to choose?
One would hope given the collegial, consultative nature of academia that a decision of such import would be taken only after having consulted with the community at large. Helger's report of EuroCrypt's business meeting sure makes it sound like the PC chair did this on her own without the broad support of the community.
seriously, if there were really mafias, don't you think that anonymous submissions would only help them? (e.g. their papers can go to their friends in the PC.)
ReplyDeleteI believe the root cause of such unfortunate "misunderstanding" is because the PC process is basically a BLACKBOX process.
ReplyDeleteWell, I think there is definitely some truth to that. More feedback and transparency would probably help clarify some of the issues mentioned in this thread. And it was mentioned already a few times at this blog that theory conferences at times give insufficient feedback.
Alas, getting more/better reviews requires more effort on someone's part (which does not mean we should not strive for it, but it means it has a price). However, there is a way to provide the authors with more feedback with virtually no extra effort: in the feedback, one can include the scores provided by the reviewers.
A few words on this. Firstly, it quantifies the decision, especially a rejection. The author gets a (partial) answer to a question: does it make any sense to resubmit to this/similar conference in the future, or is it a lost cause ? Even better feedback (although requiring some extra effort) would be to provide a few numbers, say for: quality of writing, relevance to the conference theme, originality, technical depth. That would roughly clarify broad reasons behind the decision.
All applied conferences (that I am aware of) provide such scores. I recall SoCG'05 also implemented this idea.
Piotr
By the way, does anyone know what the number of submissions to SODA'07 is?
ReplyDeleteBy the way, will this post make 200 comments? :-) I say yes!
ReplyDeleteMaybe someone can give an example of their paper that was not accepted to FOCS so that we know of this very good paper that was rejected.
ReplyDelete
ReplyDeleteBy the way, will this post make 200 comments? :-) I say yes!
You're an idiot! A post should not be judged by the number of comments it gets! The quality of the comments is more important, and some people don't even choose to post their comments on Lance's blog!
For instance, Mihai has as many comments as Sanjeev Arora, but they are not equivalent commenters!
At least comment submission is, for the most part, anonymous.
"Mihai has as many comments as Sanjeev Arora"
ReplyDeleteAh, you have been counting comments, have you :)?
Seriously, matters of importance to the community should be decided democratically, not by weight of authority. Sanjeev is a far more distinguished researcher than Mihai, and given his experience, he is in a privileged position to clarify and explain the process of STOC/FOCS selections... However, as far as making a decision on the process is concerned, the voice of one should not count more than the voice of the other.
Thankfully, we have a democratic forum where such issues can be raised and settled, the Business Meeting. If a lot of people are in favor of a certain change in the process, I hope this will actually translate to participation and action at the next Business Meeting.
I'm in favor of anonymous submissions. I have never served on a PC committee, but I have refereed many papers. The only time a a very weak paper--for which I had recommended rejection--was accepted was for a paper on which one of the authors was very famous.
ReplyDeleteSeveral times I received papers from PC members along with comments as "given the name of the authors, you can reject them very quickly"
ReplyDeleteLuca said: I think we are getting something upside-down here: it is not good conferences that give "prestige" to papers by accepting them, it is good papers that give prestige to the conferences they appear in.
ReplyDeleteI would say: a good conference is very midly affected from the rejection of a good paper (the program chair will not be fired). The author of a good paper could change his job after an unfair rejection.
ReplyDeleteAs Sanjeev said, anyone with such suggestions is welcome to raise them at the STOC/FOCS business meeting. Just note you should prepare a more serious proposal than some half-baked weblog comment, including data from the experience of other similar conference (for example CRYPTO used to have non-anonymous submissions, then anonymous submissions, and CRYPTO 06 went back to non-anynomous).
Systems conferences use a double-blind system. In some conferences the PC writes a review for accepted papers and people can comment on the paper and review (e.g. this year's SIGCOMM.)
Then again, systems people are usually considered as inferior code monkeys around here, so I'm not this counts.
Anyway, what are the good arguments for the current process? "It's the way it's always been" is not a good argument, especially considering the feelings we are seeing here.
Many papers are 'public' well before the conference deadline as they have been posted to ECCC, the ArXiv, or their authors have given talks where PC members or likely sub-reviewers will be in the audience. This is by far the biggest advantage a paper can have for conference acceptance.
ReplyDeleteEach of these can make the paper stand out from the rest because:
* the familiarity and level of understanding of the reviewers is likely to be much greater as a result. (This will tend to cause a more favorable opinion among those reviewers, independent of who the authors are, and the paper will be less easy to dismiss.)
* (in a talk) the justification for the research and the cleverness in the argument can be more easily made clear than in print.
* there might be a bit of a buzz about the paper as a result.
* if there are issues of priority, the time-stamp is clearer. There is a risk that your work will be improved prior to the conference deadline, though if your work is significant enough relative to the improvement then both may be accepted or the papers may be merged.
On a PC, the opinion of someone who is more knowledgeable about a paper is likely to carry more weight so this is a double advantage. One cannot and should not outlaw such communications; in many ways they are better than conferences at advancing the field. They do, however, destroy any semblance of anonymity (and probably contribute to some of the choices that have caused resentment that I see in these mostly very unfortunate blog comments).
This is something that anyone can do to give their papers a boost (at least until everyone tries to do it). It works in a way very similar to the usual admonition given to junior faculty prior to promotion to "go out and give talks so that you and your work will be better known".
>The author of a good paper could change his job after an unfair rejection.
ReplyDeleteI don't claim that I had an extraordinary paper nor that it was submitted to a "big" conference
(was a normal journal) but for sure was a new and interesting result and it was correct.
I had so much unjustified hassle from an anonymous referee combined with a weak editor
that when I had to choose between a postdoc position and a job as software developer I didn't
think twice. I don't regret my decision at all.
I know that TCS didn't lose anything as my results were not extraordinary
(for people that understand _just_ numbers: I've published 10 papers in journals and
conferences and my Erdos number is 3). But I also know that some (many?!) people
in TCS put their ego above scientific objectivity and this for sure is bad on the long run.
If Paul Beame's comments are meant as arguments against anonmyous submissions, they are bogus. We can still have anonymous submissions and people can still publicize their work by giving talks before submission. There are probably 400 SODA submissions. How many of the results do you know about already? Most results are NOT publicized before submission, therefore the majority of papers would likely be anonymous.
ReplyDeleteSalvatore Riina says: MAFIA ? It doesn't exist.
ReplyDeleteAs Sanjeev said, anyone with such suggestions is welcome to raise them at the STOC/FOCS business meeting...
ReplyDeleteThere seems to be a clear preference for increasing the size of the STOC/FOCS PC (I have heard stories of PC members having to review 70-80 papers, which besides being ridiculous means that these are all being farmed out to sub-reviewers; no wonder reviews are often so bad). Why does this need to be brought up at the business meeting? Can't PC chairs choose the size of the PC on their own? Does anyone disagree with the idea of increasing PC size?
Because as Sanjeev said earlier he only has $400 per pc member. however, I don't see why professors who have funds (such as NSF) can not use these to attend a PC meeting and then use the PC funds to accomodate people who don't have such funds. Then the PC size could be larger.
ReplyDelete
ReplyDeleteI know that TCS didn't lose anything as my results were not extraordinary
(for people that understand _just_ numbers: I've published 10 papers in journals and
conferences
That's not saying much without detailing WHICH conferences and journals, over what time period, etc.
A few comments:
ReplyDelete- Regarding crypto 06, and whether or not the PC chair could 'unilateraly' decide to go non-anonymous, anonymous commenter says that
One would hope... a decision of such import would be taken only after having consulted with the community at large
I think it's safe to say that the community at large is divided about this issue, many feel strongly pro or con anonymous submissions. I know the PC chair has consulted potential PC members before they joined about this (I don't know if anyone declined due to the non-anonymous submissions), and I'm sure she talked to others about this as well. Also, since there is no official policy either way, either going anonymous or non-anonymous is a 'decision of such import', and the community doesnt seem strongly behind one of these (from what I gather without a scientific study). Thus, I think it's perfectly reasonable for her to go with what she believes to be the right thing.
- It seems that here commenters are mostly for anonymous submissions. I personally am not sure about this (I see pluses and minuses of each approach), but since someone asked what could possibly be wrong with anonymous submissions, I wanted to elaborate on the reasons Jon cited, which are the ones I heard too (and I agree to one of them).
First, having the names on the submissions, puts everything in the open, and so if some PC members think a certain reviewer may have a conflict of interest with the authors, they can decide what to do about it (ignore the reviewer's comments or just take them with a grain of salt, reprimand him/her, etc). For example, in an anonymous submission conference PC, I've seen reviewers (who were PC members) arguing for papers by their friends/colleagues from the same institution. I believe these PC members truly believed that the paper was worthy (and indeed it might truly have been), and they did not view this as a conflict of interest. But if the process was non-anonymous, other PC members (myself included) would have realized the relationship between the author and the PC member, and could decide what to do with this info (in my case, I'd probably scrutinize the comments more carefully).
Second, many people believe it's ok to take authors identity into consideration. For example, when trying to assess correctness, when there is no specific suspicion, but some important parts of the proof argument are missing and just promised (which is often the case, given page limit on conference submissions). Then perhaps prominent researchers with a proven track record can be given the benefit of the doubt. Moreover, some people believe that taking the name of the authors into consideration is ok even in other instances, for example, if a paper's technical merit in terms of solved problem is considered only marginal (or even less than that), but the authors claim to provide a certain important insight into directing open problems in the research field. Some believe that if the authors have many times in the past suggested such insights that turned out to be instrumental for the field, then their paper should be accepted for this reason, while a different author writing the same paper shouldn't be accepted. Now, even if you (like me) do not believe this should indeed be so, it may be better to have this out in the open, so that if some PC members believe this and follow this, other PC members can realize this, argue back, and have the decision made more explicitly by the entire PC.
- Regarding Mafias, my impression was that most people try to really do a good job, judging papers fairly and sincerely (and in some areas, experts tend to actually be more biased against papers in their field, maybe because they understand it so well that it seems less novel and creative and ground breaking to them). However, clearly people (even the well meaning ones, who I think and hope are the vast majority), can't (and shouldn't) avoid using personal taste in deciding the merit of a paper (how interesting is the problem it solves? that is hard to evaluate objectively). Clearly, your mentors and the environment where you spent a lot of your time doing research, have an influence on what you find interesting (perhaps you wouldn't work with these people if you didn't think their approach was interesting).
An anonymous commenter said that they got many times comments to the effect of 'given the name of the authors, you can reject them very quickly'. I find that hard to believe (and it's terrible if it did happen).
- I think that reviewers should make an effort to put as many as possible of their comments as visible to authors. I've seen an instance where an author's comments were generally positive, while the comments to PC (of the same reviewer) were very negative. I guess the reviewer did not want to offend the authors with giving them the negative parts of the review. The other reviewers of the same paper gave short reviews to authors, one of which contained a typo. This caused the authors to be very upset about how their paper was treated, as they thought that it had a very good review, plus an incompetent reviewer who killed it. This was not the case at all (all reviewers were experts, no doubt even the authors would agree with that, and all unanimously agreed that the paper should not be accepted). While indeed the job of the PC is to choose a program, it shouldn't be too hard to include in comments to author all comments, except things like comparison to another submission, etc.
- About PC meetings: I liked PC meetings that met physically, because it was easier to get all present people involved in a decision for a controversial paper, and it's much easier to ask those who read the paper carefully questions, and to boil the disagreement down to its essential factors, so that other PC members can form (and voice) their opinions in a meaningful way. This is also possible in electronic meeting, but requires much more effort and good will on the part of the PC members and/or much better skills of the PC chair to orchestrate this. I generally found my involvement in papers I did not carefully read less deep in an online discussion than in a face to face discussion. I also didn't find a big problem with 'loud mouths' in a physical PC meeting calling all the shots (though I see the potential problems). The biggest problem I found with physical meetings, is that those who are not present have considerably less say in the final outcome. The best PC I served on had a physical meeting where all PC members (except one who didn't submit any reviews anyway) were present. The results of that PC seemed to have the least randomness in them of all other (not too many, but several) PCs I've served on. (As someone mentioned, inevitably there is some randomness on the marginal papers, some of which get in and some don't).
Tal Malkin
I served on both anonymous and non-anonymous PC's and prefer non-anonymous for the following (unordered) reasons:
ReplyDelete1. it is faster and easier to review a paper when you have some context (if author is an expert in the field I will focus less on verifying correctness and prior works and more on whether I think the result is important).
2. if there is a technical point I am not clear about, I like to contact the author or see if they have a full version on their home page. Of course, this can be done also in the case of anonymous submissions through the chair, but this is more cumbersome.
3. FOCS/STOC currently seem to have more of a "self-selection" mechanism in submissions, where most people refrain from submitting in the first place papers they think are not appropriate. This saves a lot of time and effort and may be related to having non-anonymous submissions.
4. Just as I like non-anonymous testing and the honor system for students, I prefer a system where we assume members of the community are trustworthy unless proven otherwise.
5. In anonymous submissions, it had happened a couple of times that I asked the author of the paper to subreferee their own paper. This is not so bad but it wastes time and also exposes the PC member assigned the paper to the author (I don't care so much but others might).
Despite this, if most people feel otherwise, anonymous submissions are fine with me, as long as they don't interfere in any way with the other important and useful ways of spreading knowledge such as posting papers on web/eccc and giving talks.
Two possible compromises are the following:
1. volunatary anonymity - authors choose whether or not to write their names. (in some sense this is the case also with so-called anonymous submissions since authors can always put their papers on eccc etc..)
2. put the names of authors at the end of the submissions, so reviewers get to first read the paper without any preconceptions, and only look to the author names at the end when they feel it's relevant.
I personally like option 2.
I forgot to sign the previous post.
ReplyDeleteBoaz Barak
4. Just as I like non-anonymous testing and the honor system for students, I prefer a system where we assume members of the community are trustworthy unless proven otherwise.
ReplyDeleteHere's the age old canard. As stated before, blind trials were implemented not to stop fraud and cheating from doctors, but to stop subconscious bias from creeping in. If a person wants to cheat a double blind won't be a deterrent.
Double blind submission is for people who acknowledge that they are human, and thus subject to bias and want to control as much as possible for that.
If a person wants to cheat a double blind won't be a deterrent.
ReplyDeleteIn this case you shouldn't object to simply having the authors names at the end of a submission, trusting the reviewer to read them only after forming an opinion of the result, to help him in evaluating correctness and relations to prior works.
--boaz
p.s. I am not sure the comparison between anonymous submissions to conference and double-blind controlled experiments in medicine is that useful.
The only time a a very weak paper--for which I had recommended rejection--was accepted was for a paper on which one of the authors was very famous.
ReplyDeleteaaaha !! there it is !!
why this bias ?
I personally attend conferences to see new innovative results, and not a famous person talking about an ordinary result.
I think it's safe to say that the community at large is divided about this issue, many feel strongly pro or con anonymous submissions.
ReplyDeleteSorry, but this just does not seem to be the case, judging from the votes at Eurocrypt.
I see couple of serious problems with the limited budget ($400 per head) and non-electronic meetings...
ReplyDelete1) Only people from drivable distance are invited. This is definitely not a uniform sampling.
2) There are very good people from other countries (Eg : India, Israel, Canada), who cannot participate in PC.
uniform sampling can solve some of the problems of bias and the formation of mafias.
In this case you shouldn't object to simply having the authors names at the end of a submission, trusting the reviewer to read them only after forming an opinion of the result, to help him in evaluating correctness and relations to prior works.
ReplyDeleteI would certainly trust them not to peek. Interestingly this is the system we use here for marking assignments. We explicitly ask the TAs not to look at the cover page when marking an assigment or exam and we trust them not to do so.
Now if you are willing to give the benefit of the doubt for a missing proof to a big name but not to an unknown, you have just created a system that gives lower marks to the junior person for equal work.
Lastly, while I think anonymous reviews would be an improvement, I don't think they would change things much. There is some bias on the system and those cases would be fixed, but it is rather minor. Its salutary effect would be the removal of the appearance of impropriety.
Sorry, but this just does not seem to be the case, judging from the votes at Eurocrypt.
ReplyDeleteMaybe it was due to peer pressure since the vote about anonymity was not itself anonymous.
Sorry, but this just does not seem to be the case, judging from the votes at Eurocrypt.
ReplyDeleteFor what it's worth, I wouldn't consider this representative of the crypto community at large. Many people did not attend Eurocrypt either because of cost/difficulty involved with travel, and other reasons.
I also get the sense (though I wasn't there) that the Eurocrypt business meeting had an atmosphere of "bashing" Crypto '06 that might have led some people to not publicly state their opinions. A private poll might have produced more even results.
If Paul Beame's comments are meant as arguments against anonmyous submissions, they are bogus. We can still have anonymous submissions and people can still publicize their work by giving talks before submission.
ReplyDeleteMy comments were not meant this narrowly. My primary goal was to present what I see as the biggest factor that causes one paper to be rated higher than another paper of similar intrinsic merit. This factor of prior publicity is much more substantial than that of authors' identities (which is causing much of the controversy here) but is likely somewhat correlated with it. My secondary goal was to give advice to authors: you too can publicize your work without passively waiting for some PC to do the work for you, so don't miss out.
I personally am not a big fan of anonymous submissions. Just one of the reasons is that they don't seem to help with this biggest factor. I concur with most of Tal Malkin's comments on the subject so I won't repeat them.
On a separate note: One of the anonymous posters suggested that the individual ratings numbers be passed back to authors along with comments. This does not seem a bad idea since it might help put written comments in perspective. (Of course these numbers reflect only initial opinions and may not reflect the full committee discussion.) From what I understand of the system typically used, this should be a matter of a little Perl script hacking.
Since my last post, various reforms have been proposed to improve the imperfections in our system. However, as many have noted, and I totally agree, these are not going to solve all the problems (including the mafia problem that I'm concerned about). Why?
ReplyDeleteObserve that, under the current system, there is very little incentive for those running the system to improve things for "us" (small potatoes, nobody, but the majority of population that submit papers). As noted multiple times, IF ONE RESEARCHER CHOOSING NOT TO SUBMIT TO A BIG CONFERENCE, DUE TO THE BIASED REVIEW PROCESS, WILL DO THE CONFERENCE VERY LITTLE HARM, BUT CAN RENDER THE RESEARCHER A MERE UNEMPLOYED LOSER. Please acknowledge the fact that WE ARE NOT LIVING IN THE IDEAL WORLD, WHERE A GOOD PAPER NOT APPEARING AT A "TOP" CONFERENCE IS AS DAMAGING TO THE CONFERENCE AS IT IS TO THE AUTHORS.
With this in mind, we can relate our situation to the various political systems we know about. In political terms, our system is certainly one where the people do not have control over the government. Good performance is not seen as the duty that this government MUST fulfill, but a GIFT to the people. Good performance will certainly buy the government praises, but bad performance is OK too, the government will still retain its power (if you don't see how this relate to our current situation, read the last sentence of the previous paragraph again?).
Currently, it seems many in our community think that this system is
perfectly fine, because those running the government are, imperfect humans, but with good intentions, nonetheless. But can we even realize that something is going wrong if it is no longer the case? Is there anything we can do when we realize that things are going wrong in our system? To me, the answer to both are, unfortunately, NO. At least, a NO for a small potato like Thomas Moore.
As I've mentioned, all the information we have about the review process are based on personal experience, what I will call single point data. This is the main reason why we can never objectively determine if the process is imperfect but fair, or being constantly abused by the "mafias". Complains about rejected submission can easily be casted as sour grape. Even if the author provides the reviews he received to prove the existence of double standard (i.e. the same BS reviews are far more than sufficient for rejecting one or more accepted papers), there is no way one can tell whether this is the norm or a rare unfortunate event. Maybe 5 out of 100 submissions are treated like this, or maybe it's 50 out of 100? Nobody can tell without a global picture.
As someone has suggested before, it'll be nice if reviews of accepted papers (if not the full review, at least the part on why this paper is recommended for acceptance) are made public. Someone claimed that this hurts the privacy of the authors - I don't see why. In contrast, this enables the readers to share the insight of the reviewers, and is
good for research community.
On the other hand, as I have proposed before, we should really start archiving the rejected submissions and the reviews they received. Of course, this must be done on a voluntary basis, but I don't see how it's going to hurt the authors of the rejected submission by providing this data
- especially if they feel they have been treated unfairly in the review process- after the related work has been published (probably elsewhere), or given up. One important issue in implementing this archive is that we have to guarantee the integrity of the input: the submissions are really the version submitted, and the reviews are really from the PC involved.
With this archive, I believe we can have a more global view of how our review process is doing. And if it is not doing well, the community can act as a whole easier, but not as many individual "sour grape syndrome sufferers" that got stoned or burnt for blasphemy.
Also, for those of you who hate Mr Thomas Moore, wouldn't this archive serve the purpose of making the entire community agree that Mr Thomas Moore is merely one suffering from sour grape syndrome, and the mafias are all illusions resulting from his mental illness? Afterall, if he still talks about the mafias without giving evidence from the proposed archive, he surely is suffering from mental illness, or at least, everybody will agree with that hypothesis.
I'm amazed this thread is still going strong. BTW, most changes proposed in the posts have been discussed by the F/S community in the past; please talk to an old-timer. And please do bring this discussion to the next business meeting in Berkeley.
ReplyDeleteThe posts of Tal Malkin and Boaz Barak (both members of the CRYPTO community, I note) seem quite representative of the views of most F/S PC members I have known.
Here's some more insight into the PC process. In FOCS'06, each PC member
was assigned up to 35 papers. There was a period of electronic discussion on contentious papers (those with two scores that differed by >= 1.5) in the week leading up to the physical meeting. I think most PC's have had similar processes. The symmetric difference between the
top ranked papers after this process and the final list of accepted is usually 20-30 (this figure based upon an informal sampling of past PC chairs). Realize that until the physical meeting, each PC member has
seen only a small sample of the submissions, so after hearing discussions on other papers PC members often downgrade or upgrade their scores.
I have *never* found the alleged Mafia phenomenon at any PC meeting. Being in a room with 20 other distinguished researchers seems to elicit uniformly good behavior. (Also, PC chairs usually will not ask known partisans or
unreasonable people to serve on the PC.) Recall also from my earlier
post that the many sources of randomness in the process (who gets asked to be on the PC, who gets assigned to each paper, the fact that each paper has 3-4 reviewers, etc.) greatly reduce the influence of any alleged mafiosi.
I disagree (based upon my most recent experience) that most committee members are indifferent to most papers. Almost every accepted paper in FOCS'06 had overwhelming support on the PC (after everybody had read the online reviews and followed the 5-10 min discussion). Mafiosi papers would also need to have passed this threshold.
I agree that famous people have somewhat of an advantage in getting papers in. Yet I have also seen instances where PC members were harsher on mistakes/omissions by famous authors than they would be on others. So it can go both ways.
p.s. Thomas Moore, I have no problem with putting all reviews for
consenting authors on the web. I am 100% confident you will not find any mafiosi pattern. I am not sure though that you understand the privacy concern. Would you rather have a CV with one fewer F/S paper, or a website with negative comments about all your past submissions available to all for perpetuity (including hiring and tenure committees)?
Sanjeev Arora
Larger PCs:
ReplyDeleteThe idea of superlarge PCs has some obvious attractions: (i) fairer representation of all areas of TCS (ii) less work per PC member.
I also see obvious downsides though: (i)leads to fragmentation of the field; authors will not bother to write papers (or at least introductions) that can be understood by the people in other subfields. (ii) how do you compare papers in different subfields? Fixed quotas seem like a bad idea. If you rank them by raw scores, then you are rewarding (and thus exacerbating?) partisanship.
(iii) how do you handle areas that are being born or are in the baby stages? Let's not forget that computational geometry, parallel
computation, database theory, learning theory, crypto, web research, quantum computation etc., were all trendy new areas at some point.
To me (iii) is the most important point, because I think the fundamental fact about TCS is that it is ever-changing, and its main role is as an incubator of research directions for all of computer science (and sometimes, for all of science).
Sanjeev Arora
Sanjeev suggested:
ReplyDeletemost changes proposed in the posts have been discussed by the F/S community in the past; please talk to an old-timer.
If there is any "old-timer" reading this blog ? If so, it would be great if he/she could describe past discussions on the ideas suggested in this thread. Which I believe were: double-blind reviewing, larger PC's, including ratings in the feedback, making reviews public for consenting authors. Sorry if I skipped something.
Piotr
Mafia?
ReplyDeleteI have been on half a dozen or so program committees, interacting with a total of perhaps 50 to 100 other PC members. Here is anomalous behavior which I have observed:
- reviews that deviated significantly from other reviews, where the PC member responsible for the odd review used some irrational or apparently (to me) biased reasons that made me suspect some unsaid hidden motivations: about 2 or 3 occurrences - granted, I tend to be trusting in general.
- PC members who systematically favored their own field without acknowledging it: about 1 to 2 occurences.
Participating in the process has actually given me great trust in the fairness of the process. Not that trendy areas are not favored, or that there is not a fair amount of luck involved for the marginal papers, or that there don't exist research areas where people tend to be highly critical of one another;s work, versus research areas where everyone is very supportive of everyone else; but I think -- I know -- that the PC members are trying to be objective and to apply their criteria of evaluation as best as they can.
Conclusion: there is no Mafia.
A reply to Mr Arora:
ReplyDeleteBefore we lost ourselves in the discussions related to the implementation details about the reform I've proposed, let's try to agree on the goal about this reform. We all see that you have very strong faith in the current system, because it is run by a group of respectable, distinguished researchers, so it should be nice and fair to everybody. And, I have no problem with that. I appreciate all the hard work done by these people.
But, are the kings and nobles in middle age respectable? If their being respectable means they can exercise their power without control, under presumably good intentions, why should we have democracy? And why would somebody argue that "matters of importance to the community should be decided democratically, not by weight of authority"? Afterall, these are good and wise people who love us, shouldn't we leave things to them? And, is the elected government under democracy respectable representative of the people's well being? If being presumably respectable, with good intention is enough, why do we still need constitutions to control the government? Why do we still need free press to monitor it? And I think the current environment in the America clearly answers the above questions?
You can call me a faithless person, but I do believe that, HUMAN BEHAVIOR THAT IS NOT MONITORED AND CONTROLLED IS CERTAINLY SOMETHING VERY DANGEROUS
AND CAN EASILY LEAD TO UNDESIRABLE RESULTS. This applies no matter how
respectable and distinguished are the humans concerned. As a previous comment has pointed out, any suggested monitoring mechanism should be viewed as an acknowledge to the fact that humans are imperfect, rather than taking this as an insult or offense.
Now, about the privacy issue you mentioned. First of all, after your elaboration, it's not that much of a "privacy" issue to me, it's merely a lack of incentive for people to reveal the reviews of rejected papers. I'm arguing about this issue with your choice of word because "privacy" is a very sensitive word. In terms of paper review, a review that said "the proposed system/algorithm has privacy problem" and one that said "it may not be a practical system/algorithm, because people may lack the incentive to implement it" will most likely lead to very different fate of the
paper.
OK, now my real reply to your comments. I guess I'm not that concerned about the "privacy" issues you raised. First of all, from the discussion in this thread so far, it really appears that hiring/tenure committees are, once again, made up of busy humans. It seems to me that they don't even have time to read all the accepted papers of the candidate, I wonder if they will go over the rejected ones and the reviews for those.
Furthermore, even if they do, it may not harm the candidate as much as you think. If one choose to reveal the reviews he/she gets for a rejected paper, I assume it means he/she is not happy with the review process the paper has undergone. In other word, he/she believes the paper should not
be rejected for the listed reasons, and is confident that it should be obvious to the rest of the world once they see the reviews, together with the submitted paper. In this case, it may do the author some good if the
hiring/tenure committee take the time to look at the revealed reviews; the committee may conclude that the submitted work should be counted as high quality work done by the author, even though it ends up at a second-tier conference. Even if the reviews point out real flaws in the submitted work, one should not feel so ashamed about it? Especially if one fixed the flaw and got the work accepted elsewhere. As many have pointed out, it's human to err, why should it be a disadvantage to one's job/tenure applications?
To further reduce the negative impact of these "bad" reviews, I suggest the implementation of a system where community members can comment on these reviews. The hiring/tenure committees will be able to see if the candidate's work has been really mistreated by the review process.
I can't believe so very few people contributing to this post see that the whole conference system in TCS is rotten and run backwards.
ReplyDeleteIMHO, one of the main problems is the bigger weight given to conference over journal publications. This issue has been raised before in this post, only to receive the ridiculous answer that "journals accept nearly everything, so they are useless to measure quality". First, it is not true that good journals accept nearly everything (try e.g. JACM). But the main and fundamental flaw in the system is that TCS conferences are run by competition and are highly selective. They shouldn't be. Let me elaborate.
Conferences such as F/S are 3 to 4 days long, so can only accomodate a limited number of talks (about 60). As a result, PCs have an upper bound on the number of "good papers" even before the submission site is open. This is simply ridiculous. You may say F/S are run by competition: "the best 60 papers get in". But with this interpretation we are automatically bound to non-objective and sometimes ill-defined measures of quality that, in addition, need be compiled in a very short lapse of time (8 to 12 weeks) and on the basis of the information given in a 10-page extended abstract with 20-page appendix that we have no time to read.
It is true that, in essentially all scientific areas, quality is measured by non-objective and sometimes ill-defined criteria such as "impact", "influence", or "interest received" of a piece of work in its field. But if we agree that we know no better measures, the point is: how can a PC calibrate the "impact" or "influence" or "interest received" of a piece of research in its field even before it gets published? We are bad at predicting the future; we already know that. And if someone is good at predicting the future he/she will be in a clear minority in the PC.
So the competition system suggested by the counterargument "the best 60 papers get in" is completely flawed. This is doing things backwards. In my view, the sensible way to measure quality would be to do it in retrospect. Every year a new committee selected by a new chair is formed twice a year to decide what 60 papers that were published in the journal issues of the last 5 years (say) are "the best 60 papers of this year". The criteria are, of course, things that can be measured in retrospect such as "interest received" or whatever makes sense. We can even imagine a submission system as long as only already published material is allowed. Obviously, journals keep their high standards of quality and refereeing system, and the current level of competition. It has already been argued in this post that the selection process in journals gives a better impression of fairness as it allows interaction with the author(s).
And what is the role of conferences in all this? Well, to publicize your results, to meet other researchers, and to set a time-stamp to your work while you wait for the journal publication to appear. Report repositories such as, ECCC or similar, can also serve as time-stamps. Run many parallel sessions to reduce competition and invite many distinguished researchers to guarantee a certain scientific level of general interest. Who are the distinguished researchers? Of course those getting a big number of "best papers of the year" in their carreer.
Haarsh T. Ries
I really agree with Haarsh Ries. Only in TCS some people (I do not say all and I do not say general CS) who think conference papers are much important than journals (and such people even do not bother themselves to send their papers to the journals for real proof-checking). We all know that in conferences, almost there is no proof-checking and as you know a theorem without a correct proof is nothing in CS or Math. As a result you see several papers in FOCS/STOC/SODA which have serious bugs. In all other sciences journal papers are much more important than conference papers, since conferences are to publicize the results and not to make a stupid competition based on some unchecked proofs, which is ridiculous. I think probabaly a good solution to all above problems about referee process in conferences, mafias, etc is to run many parallel sessions in FOCS/STOC/SODA to reduce competition amd instead give more weights to the journal publications like all other sciences. Note that it is not good also for TCS that people refer to conference papers and base their results on something which can be completely incorrect (this happened a lot in our community). Note that if you send a paper to good Math journals in which you refer to CS conferences, they usually do not buy that and they ask for a complete version or a journal version (this happened for me a lot).
ReplyDeleteHarsh Ries: TCS is rotten and run backwards. IMHO, one of the main problems is the bigger weight given to conference over journal publications.
ReplyDeleteAnonymous: Only in TCS some people (I do not say all and I do not say general CS) who think conference papers are much important than journals
Yeah, like this doesn't happen in networks (SIGCOMM, INFOCOM), databases (SIGMOD, VLDB), Information Retrieval (SIGIR) or AI (IJCAI, AAAI, ECAI).
Intelligent people can disagree about the extent, severity and ultimately the impact of FOCS and STOCS flaws. Some can argue that they are so minor that they can safely be ignored, while others can claim that they have a pernicious effect in the field. This can be a fruitful discussion.
On the other hand Mafias, "TCS is exceptionally sick", or at the other end of the spectrum "FOCS/STOCS are perfect" are positions that are out there in cuckoo land. They are more a reflection of personal frustrations for the former, and unhealthy defensiveness for the later than of anything actually taking place on the ground.
I think probabaly a good solution to all above problems about referee process in conferences, mafias, etc is to run many parallel sessions in FOCS/STOC/SODA to reduce competition amd instead give more weights to the journal publications like all other sciences.
ReplyDeleteI agree that too much emphasis is given to conferences (as most other people I've talked about this do). I do not think it is realistic to expect radical change in this regard. A gradual loosening of the publishing bar in conferences might achieve this. Now careful what you wish for. At the end of this road lies conferences such as the ICM where distinction is granted by invitation to present, and all other tracks are open for the taking. If you think the current process is open to mafias, the invited talk system is doubly so.
Note that it is not good also for TCS that people refer to conference papers and base their results on something which can be completely incorrect (this happened a lot in our community).
I've been working on this field for nearly twenty years and I've yet to run into an error in a published paper that was of any consequence. The biggest flaw I've ever found (totally unfixable, truth of result itself is in question) was, ironically, in a paper in the Journal of the ACM.
I completely agree with Haarsh Ries. CS seems to be the only area in which conferences are more important than journals. Is it possible that everyone else is wrong and only computer scientists are right?
ReplyDeleteIf you haven't run into any errors in TCS conference proceedings, you must be working in a different area.
ReplyDeleteThey are most definitely there.
I've seen fatal bugs in papers by Very Famous People. You kind of ask around "why is this paper on your CV but not on your webpage?" and you get the sheepish reply that it turns out to have unrecoverable bugs.
But hey, what can you expect from a conference system that considers itself a newspaper?
I will admit that I've even had a CCC paper with a bug that killed some side-results (most of it survives and is going to a journal... but will they review it more carefully?).
Those bugs are our own fault, not that of the reviewers... but from the grapevine I think that our paper and that of the VFPs I mentioned are just the tip of the iceberg.
So double-check on your own any result that you get out of a conference report.
And you have to wonder about the prestige CS puts upon conference publications...
If you haven't run into any errors in TCS conference proceedings, you must be working in a different area.
ReplyDeleteI haven't run into any major errors. but have ran into lots of minor errors, including some of my own (one of them a particularly nasty typo from a careless search and replace).
I'm aware of two big errors found by someone else during the course of their research. The worst part was that after contacting the authors (which in both cases were big names with lots of publications) they hemmed and hawed rather than just admitting to being wrong.
I am signing out of this thread --signal/noise ratio is too low now. I hope the inside information I gave about the STOC/FOCS PC process was useful. There is a clear need to demystify it. Note that these conferences are run by professional organizations that do have elections etc. Please do vote in them (or become a candidate yourself). Also, come to F/S business meetings.
ReplyDeleteTo Thomas Moore: realize that you have the power to get what you want. You can start a wiki website where people can post reviews they got from the F/S PCs on their submissions, and can solicit independent comments from others. As far as I know, you don't need any clearance from anybody to set up such a website. People can decide for themselves what level of "privacy" they like. Good luck!
Sanjeev Arora
The problem with errors is a significant one. Shouldn't there be some mechanism to ensure that there is a later version of the paper in which the error is acknowledged and perhaps corrected, or at least errata in a subsequent conference proceedings? This is one case where adopting a laissez-faire attitude really hurts the reputation of the community.
ReplyDeleteRe: errors.
ReplyDeleteI've heard various people suggest that any conference paper with incomplete and sketchy proofs should be given a time limit to appear in journal form, say five years or so.
If no such version appears then the result is considered unproven and first person to submit a paper containing all proof gets the journal article to their name.
It seems stupid to rank people by counting conference papers (even in STOC/FOCS/SODA) when there is a page limit in conferences and in most cases lots of proofs are just sketch and authors do not have any motivation to go through the much harder job of polishing the paper and produce the journal version which can be really checked. This is really bad for TCS to have such incomplete papers.
ReplyDeleteI don't have anything interesting to add to this thread; but, thought of taking the credit for posting the 200th comment -- just in case, Prof. Fortnow has some prize for the 200th poster :-)
ReplyDelete-Venkat Chakaravarthy