Google Analytics

Sunday, April 09, 2006

Reviewer Ethics

When you are asked to referee a paper you need to follow a set of ethical guidelines that are rarely spelled out and often ignored. Here are the rules as I see them.

The same ethical rules apply to refereeing papers or reviewing manuscripts for conferences. By "editor" I mean whomever asked you to referee or review the paper.

You should not review a paper co-authored by yourself, a member of your institution, someone you are related to, or having relations with. It is fine to referee papers by recent co-authors or by your former advisor or students. The conflict rules are not transitive, you can referee a paper by someone else at your brother's institution. If for any reason you do not feel you can give an unbiased review of the paper, discuss your issues with the editor or just refuse to referee the paper.

You should only discuss the paper with the editor. The fact that you are a referee, or even that the paper was submitted is confidential information. You should not ask someone else to look at any part of the paper without the editor's permission. You must never ever contact the authors directly.

If the paper has not yet been publicly announced, you must follow Rule Number One

Other than reviewing the paper you must ignore the paper completely for any other purpose, including your own research, until the paper appears.

If you find a simple extension or simplification of the paper: Tell the authors through the editor, they will likely add it to their paper and give you credit through a nice acknowledgment to the "anonymous referee."

If you find a significant extension to the paper: Shame on you, you have already violated Rule Number One. Best thing at this point is to wait until the paper appears and then write your extension. If the authors or someone else beats you to it, or the papers never appears, that's what you get for violating Rule Number One.

You also have put yourself in a messy situation since you are now no longer unbiased in the outcome of the paper. If you think there is a significant extension, mention the possibility in your report or keep it to yourself but don't work on it. It only leads to trouble.

24 comments:

  1. Refereeing is difficult and time-consuming. I usually read a paper enough to be able to reproduce it (or less than that). Refereeing forces you to actually read everything and be sure it is fully accurate.

    I believe that one of the benefits of refereeing a paper is that you get to think (and re-think) deeply about the results and area. This is much like in a lecture where you already know the subject: you have some dead time, in which you can ponder things that you took for granted.

    Naturally, this sometimes leads to new results. I don't think that when refereeing a paper you should handicap yourself to avoid having new things to say: The payback you get for refereeing a paper, except the gratitude of the community, is that you might make some improvements. This is natural, and I think it should not be discouraged.

    We are a friendly community, and there is plenty of credit to spread around.

    ReplyDelete
  2. I'm curious about a few things...

    Do people feel that there is a conflict of interest if you are acknowledged several times in the paper?

    I will often have received the reviewed paper independently: the authors will send me the paper to read independent of their submission, or they will have already released the paper as an institute technical report. Should any of Lance's rules on not discussing the paper apply in this case?

    ReplyDelete
  3. I disagree with Lance's interpretation of his Rule Number One. When I referee a paper, I often think about the problem before really reading the paper, in order to figure out for myself if I think the problem is actually hard/interesting. I don't see how you can really appreciate/understand the contribution of a paper unless you think about the problem first, and that might lead to an improvement.

    ReplyDelete
  4. These are good guidelines. The computer architecture community has actually done a good job of identifying these rules and making them explicit while submitting papers itself. Perhaps the TCS community also ought to do the same:

    http://www.princeton.edu/~asplos06/submission.html

    ReplyDelete
  5. It seems fairly common for professors to delegate review work to their graduate students. Is this ethical?

    ReplyDelete
  6. With respect to the ASPLOS guidelines, it's worth noting that they are stricter than what Lance suggests. For example, all co-authors for the past five years are excluded, as well as all former students forever. They also ask you to list anyone who has funded your research. In my case, I'm not actually sure who they would mean, beyond my advisor -- is that the administrator of the grant? the members of the group that decided to award the grant? someone else?

    ReplyDelete
  7. It is not ethical for professors to delegate refereeing to their graduate students. If you don't have time and think your student would make a good referee, suggest that to the editor, but it is always unethical to show anyone else a paper you are refereeing without prior permission.

    ReplyDelete
  8. There are questions regarding the acknowledgments section of a paper that you submit.

    If someone that you thank is asked to referee your paper then they could, in principle, be biased positively by seeing themselves credited. Maybe editors should never ask such people to referee to begin with, for the above reason, and because they might generally be too close to the paper. Unfortunately, such people may be technically very well qualified to referee.

    ReplyDelete
  9. Lance, I disagree with two of your rules.

    Banning reviewers from the same institution is neither workable (in cases like Berkeley, MIT, Waterloo) nor necessary (since when do people at the same institution talk to each other? :-) ). In any case, I can testify that this rule is routinely violated in practice.

    For me, Rule Number One of being a researcher is that you can never force yourself to not think about something. (That's what separates researchers from lawyers, clergy, etc.)

    If Alice proves a theorem, Bob learns about it and improves it, and then Alice fails to make her theorem public in a reasonable amount of time, eventually it becomes fair game for Bob to publish his improvement anyway (crediting Alice). Alice has a right to receive recognition for her theorem, but she has no right to hold the theorem hostage.

    ReplyDelete
  10. It seems fairly common for professors to delegate review work to their graduate students. Is this ethical?

    There is a distinction between reviewing for conference submission (in the sense of STOC/FOCS) and refereeing a final version for a journal. In the latter case the paper has almost always already appeared in some form or other. If it has appeared previously then having one's graduate student work on a referee report to check details is usually reasonable, though one should guide the student to the correct overall conclusion about the paper. (In fact, editors knowing the time commitment involved often explicitly ask for this.)

    On the other hand if one is doing STOC/FOCS reviewing then unless one is the PC member it is not OK to pass things off to a grad student. The whole point there is the level of judgement being asked of the reviewer that the student should not be expected to have.

    ReplyDelete
  11. In my experience the professor always says to the editor, for example, "I will review this, along with one of my students."

    Learning to review papers is an important part of the apprenticeship.

    ReplyDelete
  12. On the other hand if one is doing STOC/FOCS reviewing then unless one is the PC member it is not OK to pass things off to a grad student.

    Funny, what I've seen in my years in the field is almost the complete opposite. Journal papers tend to be yours alone to review, since you are explicitly given a single paper and lots of time to referee it. In contrast, for conference papers you are given a 30 to 50 papers to read and very little time to do so, so it is only natural to assume that third party help will be used.

    However as a PC you *must* read the reports from your subreferees and renormalize them according to your own scale.

    ReplyDelete
  13. Two comments:

    1)If you have received a copy of the paper independently of your status as a reviewer, you are free to work on it. Technically, you are not working on the paper you are refereeing, but on the paper that you were sent by the authors: presumably they welcome the community working on it and like to see their results used.

    There usualy *is* now a conflict of interest: you may want the paper published (or not) depending on its relation to your new derived results. At a minimum, you should inform the editor.

    2)I disagree with Scott about holding theorems hostage. If Bob learned of Alice's theorem by refereeing Alice's paper, too bad for Bob. Until the paper is published, IT IS NOT PUBLIC. Bob received it in confidence.

    The only thing Bob can do, is to appraise the editor, and ask her to let Alice know of the extension. And, (just to play Devil's Advocate)
    if Alice responds "yes, I thought of that, and am in the process of writing it up" the extension is also
    Alice's theorem.

    In practice, things usually are not that dramatic. Many submitted papers are publicly available (ECCC, conference proceedings, technical reports). Many (most?) of us are really interested in the progress of our knowledge, and are willing to acknowledge the contributions of colleagues.

    ReplyDelete
  14. So the question now is: why shouldn't submitted work be automatically public? Maybe if that was the default, people wouldn't rush to submit without working out their results properly... That is a serious problem in our community.

    ReplyDelete
  15. Clearly there is a problem when Janos' objection (to Scott) is taken to extremes: Imagine that the person decides they no longer want to publish their paper at all, or that they will get back to it in 5 years, when they have more time. Does this mean that (hypothetical-) Scott can never base any work on that paper? Can scientific or (even moreso) mathematical truth be "private"? (The answer is no.)

    ReplyDelete
  16. I agree with Anonymous above--I think all submitted papers should be public. Additionally, I think all reviewer comments should be public. That would help in the reviewing process. If you see a paper has been rejected for particular reasons and those reasons have not been addressed, then you can save the effort of re-refereeing the paper (if you agree with the reasons). It did happen to me once that I refereed a paper and found an error. The paper was rejected and the author was notified of the error. A couple months later, the exact same paper appeared in another top conference, error and all. Also, if all submitted work is public, then all submitted work will be treated equally and thus fairly.

    ReplyDelete
  17. Funny, what I've seen in my years in the field is almost the complete opposite. Journal papers tend to be yours alone to review, since you are explicitly given a single paper and lots of time to referee it. In contrast, for conference papers you are given a 30 to 50 papers to read and very little time to do so, so it is only natural to assume that third party help will be used.

    However as a PC you *must* read the reports from your subreferees and renormalize them according to your own scale.


    As a PC member you are in a role as editor so it is of course OK to ask a 3rd party. That 3rd party should have some expertise which is why I have the following to add to Lance's list:

    - It is unethical to give a conference review paper to a sub-referee in order to educate the sub-referee.

    (If one is not a PC member, it is unethical to pass on a paper given to you as a sub-referee to anyone else, but Lance already covered that.)

    For journal refereeing of a paper whose results have already appeared in unrefereed conference form, it is fair to use the process as part of educating one's graduate students.

    ReplyDelete
  18. And if a paper has been submitted to a journal, but has not been published in a conference proceedings, but will soon appear in a conference, before the 5th of the following month, and the author has given 3 talks (2 in the US) about said paper, but no graduate students were allowed to attend, then it is ok to use one's students? What if the author is himself a student, but soon will graduate, before completion of the review? Should he review his own paper? I'm confused.

    ReplyDelete
  19. - It is unethical to give a conference review paper to a sub-referee in order to educate the sub-referee.


    Educate as in "learn something about the subject" or educate as in "learn to referee"?

    ReplyDelete
  20. A gripe: how about having an identity?

    You do not have to use your name if you want, but it is hard to say
    "Anonymous 5 says, while Anonymous 8 suggests.."

    Along those lines:

    Anonymous 12, commenting on Anonymous 11 says
    I think all submitted papers should be public. Additionally, I think all reviewer comments should be public. That would help in the reviewing process.

    No, it wouldn't. The referee could be wrong. Or the author might think that the referee is wrong, or has no taste, and will want to reply. It is hard enough to get people to review papers, and having in effect a public blog about the paper would make the process even less appealing to many (and the readers of this blog are NOT a representative sample, as we enjoy reading and writing blogs)

    If an author WANTS to have this process, there are venues that allow her to do so: for example ECCC allows results to be submitted, people can comment, and authors may revise.

    As for having an incorrect result published, it is kind of risky for the author. After all, you could send email to the conference proceedings editor pointing out the mistake: many conferences then ask the author for a public correction/acknowledgement of error.

    Let me address Anonymous 11:
    Can scientific or (even moreso) mathematical truth be "private"? (The answer is no.)

    Sorry. The answer is definitely "yes." Try getting the scientific truth of the fromula or preparation method of new drugs. Or recall the story of public key cryptography: a bunch of British scientists had the concept figured out years before Diffie and Hellman. It was classifed by Her Majesty's Government.

    This illustrates the danger of not publishing: someone else may come up with the idea, and you get no recognition (do you know the name of the British scientist?)

    So authors CAN chose not to publish.
    If you are the reviewer, and there is no other source of the result, you loose.

    Of course, usually there is another
    source. Most of us are eager to let everyone know how smart we are ... -)

    ReplyDelete
  21. I agree with Lance's rules completely.

    What do you do when you mistakenly violate rule number 1 and substantially improve a result for a paper you are refereeing, but your improvement is so substantial, that you don't feel generous enough to share with the authors for an anonymous thank you? The logical thing to do if you follow rule #1 (if the original result is nice enough without your improvements) is to WANT THE PAPER to be ACCEPTED! Why? Because then when it's accepted you are free to publish your improvements! I've seen some shockingly slimy cases of people killing a paper because they've improved it and then tried to write up their improved results claiming that they never saw the original paper!

    In this situation, if you are ethical, and the original paper was nice to begin with, you hope the paper gets accepted as soon as possible, because then you can write up your improvements and cite the original.

    ReplyDelete
  22. The problem with using graduate students comes when it is hard to enforce that they follow rule #1. If the advisor is in a position to enforce rule #1, then it's good to have them learn by reviewing. I agree with the commenter who said that there's less risk with journal refereeing if the paper has already appeared in conference form, so the results are "out there". Of course the editor should be told the graduate student was involved as a routine matter.

    ReplyDelete
  23. I've seen some shockingly slimy cases of people killing a paper because they've improved it and then tried to write up their improved results claiming that they never saw the original paper!

    Sadly this is becoming more common.

    ReplyDelete
  24. Janos: Do you think it's morally permissible to publish papers based on Ramanujan's notebooks? Or were those notebooks his private business -- so tough luck for anyone who's inspired by them?

    (Of course, one difference is that Ramanujan is dead. But those who no longer publish are also "dead" in Erd�s's sense.)

    ReplyDelete