But I fear these efforts have not addressed the most fundamental question: Is the conference-publication "system" serving us well today? Before we try to fix the conference publication system, we must determine whether it is worth fixing.
In 1999, the CRA published a best practices memo about tenure that legitimizes conferences as the primary place of publication.
The reason conference publication is preferred to journal publication, at least for experimentalists, is the shorter time to print (7 months vs 1-2 years), the opportunity to describe the work before one's peers at a public presentation, and the more complete level of review (4-5 evaluations per paper compared to 2-3 for an archival journal) [Academic Careers, 94]. Publication in the prestige conferences is inferior to the prestige journals only in having significant page limitations and little time to polish the paper. In those dimensions that count most, conferences are superior.
Vardi make several points.
- Fast dissemination is not an issue with on-line archives.
- The quality of reviewing is higher for journals than for conferences.
- Every other academic field uses journals as the primary focus of publication.
Moshe ends his letter asking for further discussion.
So, I want to raise the question whether "we are driving on the wrong side of the publication road." I believe that our community must have a broad and frank conversation on this topic. This discussion began in earnest in a workshop at the 2008 Snowbird Conference on Paper and Proposal Reviews: Is the Process Flawed?
I cannot think of a forum better than Communications in which to continue this conversation. I am looking forward to your opinions.
Computer Scientists are relatively lazy, that is why they prefer conferences to journals. Conferences have higher utility and lower cost, but less social welfare.
ReplyDeleteThere was a serious divide for a while between computer science people and math people. The CS people use conferences to make their big announcements, so they would go to math conferences and get material 1-2 years out of date. The math guys would do the same to CS journals. I had heard that most people involved in the crossover have resolved their differences, with CS people reading math journals and math people going to CS conferences to keep up to date.
ReplyDeleteI think, if there is a problem with having our impact be concentrated in conferences, the solution has to be, not fixing the conferences (what would that mean? making them less selective?), but fixing the journals to make them more central to our activities.
ReplyDeleteI've been hearing from my machine learning colleagues that the situation is worse for them than in theory, in part because of stricter enforcement of journal rules that journal papers must have 30% new content compared to a journal version of a related paper: new content is required to be actually novel content that was not referred to in the older version, so the typical theoretical practice of including proofs that were omitted in the conference version isn't good enough (those proofs are still old content). By the time one has come up with enough novel content to justify a journal paper, one is well on the way to being able to use that content as the basis for a conference paper which will likely have much better impact. The net effect is that very few of their papers, fewer than in theory, get published in journals at all.
To me the system in computer graphics is better: several graphics conferences have proceedings published as special issues of journals, and the papers in these issues undergo a more stringent review (at least from my limited experience) than in theory. By making the conference papers be automatically journal papers it solves both the problem of people being too lazy to make journal versions of their papers and the problem of how much new material to include in the journal version. Of course, that has its own difficulties: one ends up with a paper that is simultaneously a conference paper and a journal paper, and (at least within the UC bureaucracy) our forms for reporting publications in tenure reviews are not set up to deal with this kind of two-headed beast.
It is worth reflecting on how we got here. For many years, the growth of high quality work in CS ready to be submitted to journals outstripped the supply of good journal pages to accommodate it. Even in the 1990's there were 2+ year delays from acceptance to publication in top theory journals such as SICOMP and J. ACM. (This does not count the 1 year+ refereeing cycles, for which we can fault each other.) It was no wonder that people didn't bother trying.
ReplyDeleteAt least we had multiple good journals to choose from. In areas such as OS, there was really only one and that hasn't changed much. Moreover, concern about "double publication" means that ACM/IEEE journals normally have some official "25% different" rule between conference and journal publication. CS theory people don't seem to worry much about that rule, but it is really enforced in other areas.
Thankfully, the ability to publish online first and a growth in the number of good journals has reduced the publication delays, at least for CS theory, so we may be in a position to re-evaluate our priorities. However, the economics of journal publication are changing, too. People are abandoning for-profit journals, reducing their role in presenting the best content despite their increased number; too many of them are of significantly lower quality than the conferences in their areas. However, societies like IEEE are also concerned about the future of their journals because online availability of conference materials and preprints makes it harder for them to break even.
I would hate to lose the excitement and energy at top conferences which is intimately tied up in their publication role. However, the better quality control of robust journal publication on the final product (for theory at least) seems an essential complement.
There is a chicken-and-egg problem for any change - we won't get the good venues unless people want to publish good work there. Abandoning the 25% different rule seems to be one way to begin to make such a change. (This would be especially valuable outside of theory; for a theory paper, the mere fact of being seriously refereed already makes it "25% different" in my book even if every symbol were the same.)
One problem with journals is that journals have typically much higher turn-around time. While this may not be much of an issue for tenured professors, graduating students and postdocs, and even asst. professors coming up for tenure, cannot really afford to spend months, and sometimes years, waiting to hear from journals about their paper.
ReplyDeleteFrom what I hear from my colleagues, this seems to be a serious issue in EE, which does have conferences, but only the journal publications are considered to be serious publications.
BTW, Michael Mitzenmacher has already pointed out some strange interactions between journal page limits and 30%-new-content rules.
ReplyDeleteAs we move more towards paperless publication, I'm seeing less justification for having page limits at all, even for conference papers.
I think many computer scientists prefer conferences over journals because of the binary decision process (in or out). In journals, reviewers have the additional flexibility to force a major rewrite (which may or may not be a good thing).
ReplyDeleteIn Australia, journal publications are essential to get ARC grant funding. The committees are typically composed of people from other fields. So, conferences will often not carry the same weight in the decision process.
I think that conferences and journals serve two related but disjoint purposes. Conferences should be venues for researchers to meet and exchange ideas. The talks and conference proceedings are supposed to facilitate this primary objective. Research results should be written up and published in archival journals after approriate and thorough refereeing giving authors a chance to revise initial submissions and so forth. The primary objective of journals is to serve as a repository of these papers (which are assumed to have been thoroughly vetted and generally accepted by the community and whose results can be used and cited in later papers).
ReplyDeleteSomewhere along the lines in theoretical computer science, these objectives got mixed up leading somehow to the absolutely nonsensical impression (amongst younger researchers and students in particular) that publications in conference proceedings have somehow comparable or even equal value as journal papers. Its time to correct this misconception as early as possible, before the community (that claims mathematical rigor as its distinguishing feature) becomes the laughing stock in the eyes of most mathematicians.
The CS people use conferences to make their big announcements, so they would go to math conferences and get material 1-2 years out of date. This might have been true decades ago but clearly no longer.
ReplyDeleteAnyone remotely interested in keeping up with any particular area of mathematical research knows that the way to do so is to browse the ArXiv mailings in his or her area -- takes \leq 10 min in the morning while sipping your morning tea /coffee, and much more convenient than making trips to conferences.
I do not see what is wrong with conferences. The turnaround time is quick, the committee is pretty good and gets credit for reviewing the paper (even sub-reviewers do). And the quality is high. Yes, we do not check every little (and not so little detail), which leads to occasional errors, but overall the system works really well.
ReplyDeleteIn contrast, journals have very long turnaround time. Also, nobody reads or cares about them. The sub-reviewers get no credit, even the editor gets little credit (aside being on the board). Why sub-review? I simply ignore any journal requests. I only submit to the journal if my paper is invited. So I do not feel obligated to review something which nobody cares about and for which I do not get any credit.
To sum up, I think journals should be abandoned all together, and progress should be made by combining conferences with on-line (unrefereed) dissemenation: ECC, EPrint, Arxiv, personal web pages etc. The only problem is: what if the author never published full version with the proofs? Well, the word would come around, and the author's future papers will be refereed much more seriously.
In any event, let's not fix the best thing we have which works - super-exciting conferences.
In contrast, journals have very long turnaround time. Also, nobody reads or cares about them. The sub-reviewers get no credit, even the editor gets little credit (aside being on the board). Why sub-review? I simply ignore any journal requests. I only submit to the journal if my paper is invited.
ReplyDeleteI hope that this attitude is not the prevalent one in the TCS community. It indicates the kind of malaise that Neal Koblitz pointed out about the publishing habits of Computer Scientists in the AMS notices a while back.
One referees journal papers not to get "credit", but as a citizenship duty to the discipline. This obsessiveness of about getting credit is very strange. One should in fact feel honored that one is asked to referee a journal submission, as it is an indications of a certain level of trust on the part of the editors. As far as reliability goes, only journal papers go through a thorough refereeing process with opportunities for the authors to improve and/or correct the paper. The STOC/FOCS review process is nothing more than a beauty-contest by comparison.
In fact, in order to bring the practices of theoretical CS more in line with those of other areas of mathematics in which it rightfully belongs, it is imperative that the worth of STOC/FOCS publications be brought down and make journals the only legitimate venue for publications.
I don't know about everyone else, but at my institution one does get (an appropriately small amount of) credit for journal refereeing. Whenever we go through a formal review on our progress for the last two or three years (or the bigger reviews for tenure and promotions) we are expected to list our accomplishments, broken down into teaching, research, and service. Refereeing counts as service and we expect to see some listed there.
ReplyDeleteTo anonymous 4:23 PM, April 26, 2009:
ReplyDeleteIt's really good you wrote this response, clearly experssing the common attitude (probably of a relatively junior researcher? it would help to know how many years post-PhD you are).
It is exactly this attitute that Vardi is worried about, which lead our area to value hastily-written papers, full of inaccuracies and mistakes, selected by a beauty contest.
The same attitute also makes journal publication so slow.
FYI: Someone already mentioned that journal reviewing counts for something; moreover, being an editor is highly appreciated by promotion and tenure committees (a lot more than PC membership).
The turnaround time is quick, the committee is pretty good and gets credit for reviewing the paper (even sub-reviewers do). And the quality is high.It works well only for straightforward incremental papers. Since the committee has no time to judge impact all they can do is judge technical difficulty (what Oded calls weight-lifting competitions). Anything outside that is usually rejected.
ReplyDeleteAs Scott Aaronson has pointed out, "conceptual" considerations are ignored in favor of technical considerations. A paper could be proposing a new paradigm which might become a new important subfield (e.g. zero knowledge, mechanism design) but all the reviewer cares about is how hard the proofs are.
I do not see what is wrong with conferences.
ReplyDeleteThis post pretty much summarizes all that is wrong with CS conferences and the attitude of (the presumably younger generation of) researchers. It is worth reminding ourselves that the final (and the only) product of theoretical research is a publication. When you are long gone, the only (one-way) communication you will have with future generation of researchers is through your publications. Do you really want to be remembered through hastily-written, two- column formatted, dubiously refereed, un-reviewed (by Math Reviews/ Zentralblatt etc.) papers published in non-archival proceedings ? If one values ones own work and takes pride in it, then one should spend time to write good journal papers for posterity -- at the same time it is a duty of every researcher
to devote some time to refereeing for journals, and doing a thorough job of it.
Finally, the strong "careerist" streak that we see more nowadays, and which is
that is reflected in this post,
is not good for the community, and in the long run not good for the careerists themselves. The earlier everyone realizes this is better for the whole community.
There seems to be three distinct main views toward conferences: established top researchers view them as a forum to talk and communicate ideas with others, ... whereas younger researchers see it as a place to introduce themselves (they are looking for jobs after all), and finally there is a group of researchers which are interested in publishing papers (for NSF, tenure, ...).
ReplyDeleteOne alleged problem is the way decisions are made about accepting papers, specially those "edge case acceptances". It seems that the first and the second group above prefer different methods. A poll of people attending STOC about various issues including this one would be interesting. The method I would prefer is to allow people who are paying the fee and putting time to attend the conference decide these edge cases. Put abstract of these papers online and allow them to vote for which paper they want to be presented.
Objections of some about the way the younger generation views STOC and FOCS seems odd to me, since the older generation by focusing on publications in these two conferences for post doc/faculty poisons have forced the younger generation to view STOC and FOCS as "weight lifting" competitions. They can change it if they really want to, by simply changing the weight they give to publications in STOC and FOCS. (The problem with time also needs to be solved.)
About completely refereed journal publications, I agree, but the time one should put for preparing, the time it takes till papers are accepted, and the time it takes for papers to be published are real problems. Mathematicians seem to be more relaxed about time, they can wait for 2 years, for CS, this seems unacceptable. Putting papers online and similar ideas will not solve this problem for the younger generation.
There have been several commenters in this discussion who have mentioned that a problem with journals is the long time that refereeing takes.
ReplyDeleteIt's worth pointing out that this isn't an insurmountable problem: indeed, there are other fields in which replies are usually much more prompt. For example, a number of top physics journals (Physical Review Letters, Science, Nature...) usually reply in a matter of weeks.
I've never understood the year+ turnaround times for submissions to CS journals. Why agree to referee a paper if you know you're just going to sit on it for months?
There seems to be three distinct main views toward conferences: established top researchers view them as a forum to talk and communicate ideas with others, ... whereas younger researchers see it as a place to introduce themselves ...I fully agree with the views expressed in this post. There are two main points being made and I want to reiterate them in a practical form.
ReplyDelete1. The evaluation of TCS researchers should be delinked from the number of FOCS/STOC papers. At best having a paper accepted at one such conference should be counted as equivalent to an "invited seminar talk" indicative of being research active, but never as a publication. Senior people in personnel and tenure committees have a role to play here.
2. Journal turn-around times should be reduced to at most one-year. This should be possible if the editors make it a high priority. At the same time we could consider increasing the number of pages published per year in mainstream journals taking into account the increased level of activity in the field.
To what might we attribute the community ambivalence and slow pace of journal refereeing?
ReplyDeletePeople are too selfish and career-oriented to spend the time on refereeing.This is not true. Hundreds of researchers are very willing to spend significant amounts of time writing multiple sub-reviews for major conferences, even though they do not get the boost from the PC membership. The reward from the listing of their names in the front of the proceedings is hardly compensation for their time spent.
There's no rush because the paper will take two to three years to get published anyway.
I think that this is a significant part of the attitude problem, though it is an out-dated view. Journal publication now is generally shortly after acceptance.
People might have negative attitudes towards certain for-profit publishers. The number of pages in good journals now in society, online not-for-profit publications, and other publishers who are 'good actors' has grown substantially in recent years.
It requires a level of concentration and care beyond conference reviewing, a level that our community lacks.
I admit that it does take more time per paper but reviewing one paper compared to the time I need to set aside to sub-review multiple papers for a conference really isn't a problem.
A belief that the timing of the report and even the publication itself doesn't really matter to the authors or editor of the paper under consideration. I think that this is real and is the biggest issue faced by journal publications in getting referee reports. Helping out a PC member who is under a large workload and hard time constraints is an obvious favor. People also know the importance of the conference decision to the community and the value of an acceptance to the authors.
There is little to match the notion of a PC meeting deadline to enforce the timing of journal reviews and do a favor for the editors. The one circumstance that has come close to matching it in my experience is that of editing a conference special issue with a hard deadline. (This was for the CCC08 special issue which has to be produced in time for distribution at CCC09.)
If journal publication had some sort of guaranteed short time-line after acceptance then it would encourage quicker reviews.
There is now less of a feeling about doing a favor for the editor - growth in the automated systems for referee requests has made the request process more impersonal. I admit that I ignore such requests when they come from publishers whose actions I disapprove of. Previously if it came directly from the editor I would feel more of a sense of obligation.
The biggest problem in giving some priority to journal refereeing is that we don't seem to think that the journal publication itself (or its timing) makes much of a difference to the authors or the community.
How do we resolve this? The Godel Prize was set up in major part to try to reward journal publication but it is a "test-of-time" award and not likely to encourage quick turnaround for journal publication. (14 years is a long time). Conference special issues seem to generate suitable attention but the rest of journal publication languishes a bit.
It is up to the journals to fix this with incentives: Journals could set up their own prizes and rules for papers that they publish outside of special issues. To encourage real timeliness in their offerings they could set up much shorter deadlines for eligibility between conference and journal publication. There doesn't have to be money involved - the prestige itself might be enough.
If everyone knew of the stakes their behavior would be very different.
Paul Beame wrote ...
ReplyDeleteJournals could set up their own prizes and rules for papers that they publish outside of special issues. I agree with the rest of the post,
but I think this is a bad idea. No journals worth its name in mathematics (for instance) offers such prizes. The incentive of publishing a papers in a journal should be the author's own conviction that his or her results are worth preserving for ease of use of future researchers. That should be the *only* incentive -- not that cushy job in a private collge, or points towards tenure, or getting funding from NSF etc.
If this conviction is missing from the majority of active researchers in theoretical CS, then sadly the conclusion must be that most aspects of CS are "engineering oriented" and necessarily ephemeral in nature, and the authors of papers in these areas do not feel that their work is worth spending the time preparing a careful journal paper. I for one believe that some significant part of theoretical CS is fundamental enough to worth writing journal articles. But in fact, for these parts publishing in journals (math journals mostly) is already the accepted norm -- so things are probably where they should be.
The incentive of publishing papers in a journal should be the author's own conviction that his or her results are worth preserving for ease of use of future researchers.
ReplyDeleteIn our current framework of society publications, major conference publications, though flawed, are more accessible than many journals.
No journals worth its name in mathematics offers such prizes. There are other more useful models than mathematics journals.
My model was the IEEE Information Theory Society which gives an annual award to a paper published in major journals during the previous calendar year. This paper almost always was published in IEEE Trans. Information Theory which is why I viewed it as a journal award but your are right that this is not really accurate. There is a similar joint award with the IEEE Communications Society. Those journals have very fast turnaround times unlike CS journals.
It is up to the journals to fix this with incentivesAlternatively it could be up to senior, tenured researchers who no longer need the prestige currently associated with conferences to turn things around. This could be done by, say, only submitting to journals and abandoning conferences. Once prestige shifts away from conferences toward journals, journal reviewing will get faster, and the community will be better off for it because we'll have published results which are actually checked for correctness. Conferences do provide advertisement of new results, but cheaper infrastructure already exists to do this: arXiv/ECCC.
ReplyDeleteOne problem is that senior TCS researchers have succeeded to convince T&P committees that STOC/FOCS papers count as journal publications (and even more).
ReplyDeleteTruth is: When I see a submission that has previously been published in a good conference (not just STOC/FOCS, even the next tier), I tend to believe that it is interesting, and worth publishing if it is correct and well-written.
It is more likely that a review procedure will be started for such papers. (Not all papers pass the bar to even get a full review.)
But reviewing for corretness and presentation often reveal major problems with these papers.
As a side note: Many good researchers still publish their work in journals. E.g., the last issue of JACM has papers by Arora, Rao, Vazirani, Raz, Kahanna, Klein, to name a few.
(Many of these papers previously appeared in STOC/FOCS/SODA/etc...)
JACM, by the way, publishes papers approximately 2-3 months after acceptance, shorter than many conferences.
This could be done by, say, only submitting to journals and abandoning conferences.
ReplyDeleteThis presupposes that abandoning our current conferences would be a good thing for our field. I couldn't disagree more. Some may decry research driven by conference deadlines but these are huge positive factors in the progress of our field. We do also need more of the detailed and careful reviewing from strong journals. This growth should not come from tearing down our conferences but rather from building up our journals.
Re: turnaround times for journals.
ReplyDeleteAllow me to give one data point. I submitted a paper to a well regarded journal in 2004. The paper was accepted, with a request for minor revisions, in 2006. A revised version was submitted in the beginning of 2007. Two years later(!), I have yet to hear back regarding the status of the paper.
(I do not claim that this is representative, but the fact that there is no upper limit to the process is certainly a disadvantage of journals over conferences.)
"Alternatively it could be up to senior, tenured researchers who no longer need the prestige currently associated with conferences to turn things around. This could be done by, say, only submitting to journals and abandoning conferences. "
ReplyDeleteI completely disagree, they are not submitting to these conferences for prestige, they are presenting things that other people want to hear. Destroying the prestige of conferences will neither solve the problem for journals nor other problems. The view that conferences are just about prestige is not correct. They are started for these aims.
correction: They are started for different aims.
ReplyDeleteOn the comment: JACM, by the way, publishes papers approximately 2-3 months after acceptance, shorter than many conferences.
ReplyDeleteAnd from my experience: JACM accepts papers approximately 2-3 years after submission. That is, the refereeing is the bottleneck.
I just read Vardi's editorial. He makes a strong case, and I think he is largely right.
ReplyDeleteI do not dispute the advantages of conferences, especially rapid dissemination of results. But why can't we have both conferences and a stronger journal culture?
I think it’s great that Moshe has started this debate. Many good points were made. I actually see a lot of consistency between the various points made so far: the anti-conference side says that conferences do not encourage high scientific quality, and the conference advocate says that that may be so, but publishing in conferences is good for my career. I’d like to raise a few more problems with conferences in addition to the quality ones mentioned. Basically, they cost a lot in terms of stress, time, and money.
ReplyDelete1. Stress and Deadlines. A regular conference can easily bring four deadlines or more. First, there is the rush to get the submission done. Lawrence Saul has written a hilarious spoof of the craziness of this, where he describes his race to submission in the style of the 24 show http://cseweb.ucsd.edu/~saul/icml24.html . If the paper is accepted, there will be a tight deadline to prepare the camera-ready copy. Often this comes with copious instructions, forms to be filled in --- you can’t actually work on polishing your paper because of all the paperwork the organizers want. Then there is a third deadline to get the actual presentation ready. It’s quite possible that you are meanwhile serving on one or more program committees. While you are dealing with your own papers, you are supposed to review 6 or more others. Then you are supposed to read the authors’ replies to your reviews and/or send lots of e-mails to your fellow reviewers to discuss your views. All this feedback to feedback is a well-intentioned attempt to improve reviewing quality, but it takes a lot of time that could be spent on research instead.
2. Money. Just the conference fee can be $500+. If you actually go, pay for plane ticket, hotel etc., it can easily cost you another $1500-2000. So we are talking $2000 just to get a paper published. There are many places where people just don’t have that kind of money. Certainly not for the 4+ papers a year that are pretty much expected at research institutions these days. Do we really want a researcher’s ability to present their work to the community to depend on how much they can pay? I have noticed a trend that people from some places, like China, just don’t show up to present; the reason given is often a Visa problem. They usually still have to pay the fee to register. At this point we reach absurdity: the researcher is not present because they don’t have the means to come, the program is in disarray, and the researchers were effectively charged a fee for publication.
Why things won’t change. I am pessimistic that things will change despite the best efforts of the ACM and CRA. I think that speaking in terms of game theory, we are at an equilibrium state: the people who have the power to change the state of affairs would not benefit from a change, and th people who don’t have the power to change it have no choice but to go along. Let me explain a bit more.
ReplyDeleteBasically the conference system is a very resource-intensive way of disseminating scientific research, as I explained above, mainly in terms of the researchers’ time and grant money. So to get scientific credit and recognition through conferences, you need a lot of resources, which means that the scientists at the top departments with the best resources have a headstart, which in turn gets them more resources, etc.. This is a well-known phenomenon in science dubbed the “Matthew effect” http://en.wikipedia.org/wiki/Matthew_effect, and it happens not only. However, the conference system exacerbates the Matthew effect. Now I’m not saying that senior faculty at top institutions are engaged in a big conspiracy to protect their standing. It’s just that they don’t feel the problems as much, because they can download a lot of the work and stress that go with conferences on to students. So for faculty at top institutions, the system is working pretty well: it gives them a high publication count, and a lot of the stress and grunt work is downloaded to students.
From the students’ perspective, they are caught in the logic of the arms race: even if they realize what is going on, they have to chase the publication count in order to survive academically. I have on several occasions advised a student that if they did another experiment or another proof, their paper would be really good, and the response has been that they think the reviewers will accept it anyway. Perhaps, I say, but don’t you want to take pride in your work? Then the answer is that they cannot afford to take the time to make it good because they need at least one paper a year to have a chance on the job market. I don’t know what to say to that.
So it doesn’t help to keep pointing out the problems with conferences unless you change the situation of individuals so they can afford not to publish so much there. And unless you change that, the conference publication will be considered the desirable competitive achievement that brings you recognition, so you try to get in there, etc.
Another point about quality: a concern about conferences has been that they discourage originality and innovation, especially conceptual innovation. This problem has been nicely described by Kenneth Church (“Reviewing the Reviewers” rakaposhi.eas.asu.edu/church-precision-reviews.pdf) and the chair of the Computing Research Association on publishing quarks or LPUs (least publishable units) http://www.cra.org/CRN/articles/march09/Reed-Musings.html .
ReplyDelete