A review of the usual referee process: The editor-in-chief of a journal receives a submission and assigns a member of the editorial board to act as editor for that paper. The editor will contact potential referees and ask them to review the paper. The referees read the paper and write a report on whether to recommend the paper for publication and give suggested changes. They send this report to the editor who relays this information to the authors without revealing the identities of the referees. The authors will comment on the referee reports and revise the paper. The paper often goes back to the referees and the process repeats until the editor is happy with the paper or decides the paper is not worthy of the journal. The editor then sends his recommendation back to the editor-in-chief.
An argument for posting the reports goes as follows: If we make the reports public (though still anonymous) referees will feel a greater responsibility to make their report thorough and accurate.
The process of refereeing requires considerable back and forth conversation between the three parties: the authors, the editor and the referees. Posting the original reports will give a misleading view of the process and will cause referees to act far too cautiously about pointing out problems in a paper.
We have an editor responsible for making sure that he has enough quality reports to properly make a decision about publication. Using the public as an editor will often lead to some very difficult and messy situations.
Keep in mind the distinction between the referee process and reviews of published papers. We should encourage publicly available reviews of research papers whether in Mathematical Reviews or informal settings like weblogs. These reviews help us find the gems among the sea of research papers out there.
I thoroughly dislike the idea of making referee reports public. The proposal to make them public somehow stems from a basic mistrust of the exisiting authors-editors-referees path of communication. I have no such mistrust. For the purposes of evaluating relevance or importance of a paper to a researcher, as Lance has suggested, something like MathReviews is very helpful and we should encourage such venues. As a referee, publicizing my reports (even anonymously) would seriously hamper my objective evaluation of a paper. This is especially true if the report includes controversial comments about re-discoveries, earlier history (e.g. previously rejected from another journal or conference) of the same paper, etc. Of course, such things can be edited by editors, but how much more do we want to burden our already overloaded (mostly volunteer) editors?
ReplyDeleteI agree that reports should not be publicised (and I've said as much). One nitpick: the back-and-forth you describe is for journal refereeing. In conference refereeing, there is no right of reply for the author. On the other hand, a referee who recommends "accept conditioned on revisions" may have no mechanism for ensuring that the author actually carries out the revisions. So conference refereeing is not the same sort of reasoned back-and-forth you describe in this entry. There may be discussions within the PC, but in the cases I've seen the author is not involved except in extraordinary circumstances. I don't think this makes posting conference referee reports a good idea, but it does highlight a source of concern.
ReplyDelete-David Molnar
People that sign their referee reports evidently reveal not only their own identities but also some information about the identities of other referees. Sometimes very important information.
ReplyDeleteI think the "moral arguments" for anonyminity that are being given are great. But let me suggest a more practical reason. How many journals can you name where the referee process is not already too long and too slow? Part of the reason for this is it is often very hard to find people willing to review papers.
ReplyDelete"An argument for posting the reports goes as follows: If we make the reports public (though still anonymous) referees will feel a greater responsibility to make their report thorough and accurate."
Actually, it would make me feel less inspired to referee. Unless I tried to obfuscate my style, I imagine the public could conclude with high accuracy what referee reports were mine. Perhaps this would make me feel a greater responsibility to do a thorough job; more likely it would make me feel like the burden of refereeing was even greater, and I would say no more often.
Having referees sign their report is a terrible idea. Just imagine a graduate student refereeing a paper by a senior person who might some day decide about her job application.
ReplyDeleteI don't like the idea of making the reports public either, but I think the main argument for this suggestion is that it helps the referees not waste time on papers that get submitted many times to different conferences until they find their place in the "food chain".
--Mohammad
I occasionally get sent papers to referee. I always decline, since I've never once been the primary author of an academic paper and have only a vague idea of how they're judged, although I do as a courtesy read the paper and give a few comments on it.
ReplyDeleteSince reviewers are anonymous, I could just write completely incompetent reviews and rely on the anonymity to avoid soiling my reputation, but that would appear to be a bad thing.
-Bram
I was thinking about the web-log format:
ReplyDeleteWhy not to publish our papers on-line, where people can comment on it / discuss it, as what we do with Lance's Posts?