The ways we communicate scientific research has vastly improved, particularly with the advent of the Internet, but the need for that basic mission will never go away.
Noam Nisan laments that journals do not provide a quick form of dissemination or do the proper amount of vetting. He's correct on both points. Computer scientists need to take journals more seriously to improve the vetting process and the speed to publication. But also journals have never played the role of quick dissemination in computer science. That role has been taken by conferences, departmental technical reports and more recently on-line archives. Journals don't compete with sites like ArXiv, they play different roles.
Tim Gowers suggests a commenting/scoring system for reviewing papers. I'd love to see such a system built into ArXiv and ECCC. But it won't supplant the need for academic journals. Most papers won't get reviewed and most researchers won't review papers. Collaborative projects like Wikipedia, Polymath, Math Overflow (and the TCS descendant) are incredible resources but just a small number of researchers are significantly involved. If you reward people for reviewing papers (through reputation or otherwise) then people can decide not to review guilt free.
We are moving to a world where we rank research papers not on where they appear but by how many citations they achieve, a statistic the Internet has made easier to measure. One can cite an ArXiv paper just as easily as a JACM paper. The incentives for an author to do more than throw up a paper on an on-line site are going away. We will no longer fulfill the mission of journals and future scientists will struggle understanding the how and why of what we did. Is this the gift we want to leave to the next generation?
"But it won't supplant the need for academic journals. Most papers won't get reviewed and most researchers won't review papers."
ReplyDeleteThe most commonly mentioned reason not to have such an online review site is that there could be mean/false comments that the author would have to defend. However, if ever such a site (hopefully) comes to fruition, then most papers will most likely receive zero comments, i.e. no one is interested in them. This will make your paper look worse then if someone actually comments on it!
One reason people currently write mean reviews is that reviews are not posted. Even if all conference reviews were posted anonymously, I don't think people would write some of the stuff they write now. Also, the fact that all of our reviews are not released hides a lot of information about the quality of reviewing. If all reviews were public, people would be less incentivized to submit/make public mediocre papers but we would also be able to see patterns of favoritism or unfair treatment of certain areas.
I think currently there are also mean reviews, because sometimes a paper is simply mediocre. In the case of an online review system, it would be ignored. But since you are reviewing the paper for a top journal, you have to write something that sounds mean in order to ensure that your opinion is clear. However, in the online scenario, you could simply write nothing at all. Another case when there would be mean reviews is when a paper is over sold. However, in this case, time will tell if the mean reviewers were too mean.
What concerns me about the whole "throw a paper up on an on-line site" model is that it then falls to the authors to advertise the paper, and some will be much better at that than others independent of scientific/technical merit. It seems to me the papers that will be most cited will be the ones whose authors have the most active Twitter/Facebook presences.
ReplyDeleteI recently saw someone suggest on Twitter that a good measure of paper interestingness is the number of tweets made during its presentation. I sure hope that wasn't a serious proposal.
I think the two ideas of keeping journals running and introducing online repositories with open comments and reviews are not mutually exclusive. What I'd like to see is people still submitting to journals, but with every submission doubling as a new entry in an open repository. This way, editors and reviewers of journals could still guarantee a certain vetting and a number of (hopefully) high quality reviews. On top of that, the submissions will be open to outside comments and reviews, which should counter-balance the cases of poor reviewing or short-sightedness of the journal's reviewers. This can go both ways: a paper may be rejected with nasty reviews, but still collect positive outside reviews, an indication that the reviewers were probably prejudiced. Or, conversely, unsubstantiated positive official reviews from the journal may be countered by open reviews that show fallacies and significant shortcomings overlooked in the official publication. This would hopefully create a feedback loop that makes journals more precise in their decisions.
ReplyDeleteNoam's correctness has nothing to do with computer science, or its supposed need to "grow up", as you so insultingly put it. His post was instigated in part by grumblings by mathematicians that the mathematics journal system is broken.
ReplyDeleteMath journals are no longer an effective vehicle to disseminate mathematical results; arguably, that was true even *before* the ArXiv. Mathematicians raise exactly the same complaints as computer scientists raise about slow, mean-sprited, and sloppy reviewing.
The incentives for an author to do more than throw up a paper on an on-line site are going away. We will no longer fulfill the mission of journals and future scientists will struggle understanding the how and why of what we did. Is this the gift we want to leave to the next generation?
Why not? It works for physics.
"One reason people currently write mean reviews is that reviews are not posted."
ReplyDeleteIf you think people won't write mean things if they are posted, you haven't been following the comments in this blog too long.
"What is the purpose of an academic journal? To provide a permanent vetted record of a specific research endeavor."
ReplyDeleteLook up "Algebraic methods for interactive proof systems" (FOCS 1990, JACM 1992). It is twenty years old and IEEE wants $30 to download it, ACM $15. They'll charge me to get even my own papers. These organizations treat us all with scorn. They will not change unless better alternatives force them to.
"If you think people won't write mean things if they are posted, you haven't been following the comments in this blog too long. "
ReplyDeleteI don't think so for the following reason. While the comments on this blog are sometimes nasty, they could be worse. Meanwhile *most* of my papers that eventually get published still get nasty reviews.
Consider one of the most famous nasty comment incident on this blog regarding jobs. People were saying (in very nasty ways) that so-and-so only got the job b/c of x y and z. However, even *if* that were the case, the nasty comments showed that x y and z (in this case sexism) still exists, so it was a legit reason to make a hiring to combat it and rectify the situation.
Same with public comments. Let's say I go out of my way to trash someone's paper for reasons that have nothing to do with the quality of the paper. People will read those comments, see that they are irrelevant, uncharacteristically harsh or what not, and they will see that this paper/person/topic is being unfairly treated. Such nasty comments stand out in a public forum. Right now, since our feedback is mostly private, they are simply the norm.
I suggest the following procedure.
ReplyDelete1. We stop publishing in traditional media (no more journals, and conferences become just a place to meet, not to publish).
2. A new paper is posted on the web with possibility to get comments.
3. If you use a paper substantially (i.e., use a theorem or a definition from it), you *must* comment on it unless the paper already received three positive comments. Of course, that means you have to verify the theorem you use :-)
4. There is an index of papers that received enough positive comments. Of course, other classifications must be available as well.
5. It must be normal that you receive a paper from the author, asking to comment on it. You must treat it as a service to the community just as now when you receive a paper for refereeing.
6. In case anonymity is concerned, you can also send your paper to "editorial boards" so that an anonymous member of such a board comments.
I don't understand the requirement that someone *must* comment on a paper. This is where comments might become nasty: someone who does not care/know about the paper is forced to comment on it.
ReplyDeleteIf a paper has no comments, that says a lot about the paper. Why not just leave it at that?
You seem to be unaware that at least one such "online reviewing" project is under way: http://groups.google.com/group/scirate. Currently its development is in a very early stage, so you could contribute a lot by discussion, sharing ideas etc. (the previous version of scirate was run by Dave Bacon and used mainly by quantum information/quantum computing people).
ReplyDeleteMath journals are no longer an effective vehicle to disseminate mathematical results; arguably, that was true even *before* the ArXiv.
ReplyDeleteThis is true, but beside the point. The way math publishing has evolved is that the arXiv is for rapid, widespread dissemination and journals are for ranking by prestige and for permanently archiving polished, refereed versions of the paper. In the 90's, there was some expectation that journals might disappear or radically change once their role in dissemination was no longer needed. However, it doesn't seem to have played out that way. It turns out that once you solve the dissemination problem, nobody really cares whether the dissemination happens through journals or not.
Mathematicians raise exactly the same complaints as computer scientists raise about slow, mean-sprited, and sloppy reviewing.
As a mathematician, I disagree. The computer scientists I talk to complain much more about journal refereeing, and I suspect the problem actually is worse in computer science. Because CS journal publication hardly counts for anything except eligibility for the Goedel prize, there is less pressure to do a good job of refereeing. Plus conference reviewing sets a terrible precedent of sloppy and fad-driven refereeing; in principle, the bad example wouldn't have to extend to journal refereeing, but in practice, it's hard to raise the bar for what constitutes reasonable reviewing after the important decisions have already been made at the conference level.
Anonymous @4:42pm "I don't understand the requirement that someone *must* comment on a paper. This is where comments might become nasty: someone who does not care/know about the paper..."
ReplyDeleteNo, I said you must comment *if* you use the paper. In this case you certainly know and I would assume you do care...
Michal Kowowski: "You seem to be unaware that at least one such "online reviewing" project is under way: ...scirate"
Is it just a discussion on a software tool. What we really need is to change general philosophy and policy.
I don't know how journals will change to accommodate the invention of the internet. The journal as the vetted, recognized endpoint of good science with online dissemination and discussion via communities like Arxiv seem like a good bet. The journals do charge an arm and a leg for access and I think that will end now that the internet has broken their monopoly on access and distribution of papers. Its funny how many times I have found a paper via google search that was offered for $$ through the traditional journals. Anyway, it is early in the internet revolution, so who really know? (See Clay Shirky's article "Thinking the Unthinkable" for a discussion of the social implications of this revolution).
ReplyDeleteWhat is also interesting is how these changes impact the people doing science in the trenches. Does it ameliorate or aggravate the "publish or perish" pressures in academia? Does it reopen some areas of science to amateur or non-professional contributions, as it was in its early history?
Noam Nisan wrote in his post "Journal system ... has created a lot of harmful side effects: the writing of countless worthless papers, lack of recognition for surveys, books, or other non-”paper” contributions, blind and silly use of metrics like impact factors for hiring, grants and promotion which lead to wasteful optimization of these rather than of real research."
ReplyDeleteI completely agree with all this diagnosis. But I am afraid that NOT journals (or at least not only they) are guilty for all this mess. This is our scientific system as such. It is the system which forces people to publish at "half of the road". It is hiring rules which force young people to publish instead of spending one more year in a depressive search of the truth. As A. Einstein once said: "Not everything that can be counted counts, and not everything that counts can be counted." Our scientific system is NOT consistent with this axiom. It counts all the mess (which Noam mentioned), but not a real research impact. I am not sure that this can be changed by introducing a crowd-sourcing research: the system will then count even more mess as "research impact". Journals, however, could be repaired: it would be enough that editors of respectful journals leave these fat guys (publishers) and create equally respectful electronic journals. This alone will not remove all the mess in the system, but at least will make it cheaper ...
We all seem to assume something special about the role of journals in maintaining publication standards with things like anonymous peer review. This seems to be relatively recent.
ReplyDeleteI recently read John Moffat's autobiographical "Einstein Wrote Back" where he discusses Einstein getting very upset in the 1940's that the ways of science had changed - now some anonymous person was suggesting changes to his paper before publication (something he viewed as a rejection). (Moffat himself had somewhat of an axe to grind on this score - he blamed conservative peer reviews for preventing some of his own work on extensions to general relativity.)
The Wikipedia article on peer review also points out that though it was common in the medical field earlier, the formal journal review process really only took hold in the rest of science in the mid-20th century.
@Paul: I am also not happy about "anonymous peer reviewing". This is something like a side effect of the system itself. But we forget that papers are (or should be) accepted/rejected by EDITORS, not by the reviewers. This should be also changed. Reviewers are NOT anonymous for the editors. But they should only be treated as expert opinion helping to formulate the decision. Nothing more. Decide should she/he (handling editor). These are editors who make the difference between journals, not "anonymous referees". In a "holly world" I could even imagine journals run by some prominent scientists deciding by *themselves* what to put in their issues.
ReplyDelete@Stasys, I don't know if we live in a "holy world" but that is already happening:
ReplyDeletehttp://www.wisdom.weizmann.ac.il/~oded/my-choice.html
Personally I found a paper mentioned in Oded's my-choices more credible than a paper that is published in even our sacred conferences STOC/FOCS or top journals.
Ideally Gower's suggestion will lead to a centralized system where every researcher will have an account and can write short comments and summaries of papers they look at or read. Then we can go and check the papers that our favorite top researchers in each area found interesting.
I wish we could upvote comments here, because I would upvote Stasys's first comment above.
ReplyDelete