Sunday, April 09, 2006

Reviewer Ethics

When you are asked to referee a paper you need to follow a set of ethical guidelines that are rarely spelled out and often ignored. Here are the rules as I see them.

The same ethical rules apply to refereeing papers or reviewing manuscripts for conferences. By "editor" I mean whomever asked you to referee or review the paper.

You should not review a paper co-authored by yourself, a member of your institution, someone you are related to, or having relations with. It is fine to referee papers by recent co-authors or by your former advisor or students. The conflict rules are not transitive, you can referee a paper by someone else at your brother's institution. If for any reason you do not feel you can give an unbiased review of the paper, discuss your issues with the editor or just refuse to referee the paper.

You should only discuss the paper with the editor. The fact that you are a referee, or even that the paper was submitted is confidential information. You should not ask someone else to look at any part of the paper without the editor's permission. You must never ever contact the authors directly.

If the paper has not yet been publicly announced, you must follow Rule Number One

Other than reviewing the paper you must ignore the paper completely for any other purpose, including your own research, until the paper appears.

If you find a simple extension or simplification of the paper: Tell the authors through the editor, they will likely add it to their paper and give you credit through a nice acknowledgment to the "anonymous referee."

If you find a significant extension to the paper: Shame on you, you have already violated Rule Number One. Best thing at this point is to wait until the paper appears and then write your extension. If the authors or someone else beats you to it, or the papers never appears, that's what you get for violating Rule Number One.

You also have put yourself in a messy situation since you are now no longer unbiased in the outcome of the paper. If you think there is a significant extension, mention the possibility in your report or keep it to yourself but don't work on it. It only leads to trouble.

20 comments:

  1. Refereeing is difficult and time-consuming. I usually read a paper enough to be able to reproduce it (or less than that). Refereeing forces you to actually read everything and be sure it is fully accurate.

    I believe that one of the benefits of refereeing a paper is that you get to think (and re-think) deeply about the results and area. This is much like in a lecture where you already know the subject: you have some dead time, in which you can ponder things that you took for granted.

    Naturally, this sometimes leads to new results. I don't think that when refereeing a paper you should handicap yourself to avoid having new things to say: The payback you get for refereeing a paper, except the gratitude of the community, is that you might make some improvements. This is natural, and I think it should not be discouraged.

    We are a friendly community, and there is plenty of credit to spread around.

    ReplyDelete
  2. I'm curious about a few things...

    Do people feel that there is a conflict of interest if you are acknowledged several times in the paper?

    I will often have received the reviewed paper independently: the authors will send me the paper to read independent of their submission, or they will have already released the paper as an institute technical report. Should any of Lance's rules on not discussing the paper apply in this case?

    ReplyDelete
  3. I disagree with Lance's interpretation of his Rule Number One. When I referee a paper, I often think about the problem before really reading the paper, in order to figure out for myself if I think the problem is actually hard/interesting. I don't see how you can really appreciate/understand the contribution of a paper unless you think about the problem first, and that might lead to an improvement.

    ReplyDelete
  4. These are good guidelines. The computer architecture community has actually done a good job of identifying these rules and making them explicit while submitting papers itself. Perhaps the TCS community also ought to do the same:

    http://www.princeton.edu/~asplos06/submission.html

    ReplyDelete
  5. It seems fairly common for professors to delegate review work to their graduate students. Is this ethical?

    ReplyDelete
  6. With respect to the ASPLOS guidelines, it's worth noting that they are stricter than what Lance suggests. For example, all co-authors for the past five years are excluded, as well as all former students forever. They also ask you to list anyone who has funded your research. In my case, I'm not actually sure who they would mean, beyond my advisor -- is that the administrator of the grant? the members of the group that decided to award the grant? someone else?

    ReplyDelete
  7. It is not ethical for professors to delegate refereeing to their graduate students. If you don't have time and think your student would make a good referee, suggest that to the editor, but it is always unethical to show anyone else a paper you are refereeing without prior permission.

    ReplyDelete
  8. There are questions regarding the acknowledgments section of a paper that you submit.

    If someone that you thank is asked to referee your paper then they could, in principle, be biased positively by seeing themselves credited. Maybe editors should never ask such people to referee to begin with, for the above reason, and because they might generally be too close to the paper. Unfortunately, such people may be technically very well qualified to referee.

    ReplyDelete
  9. It seems fairly common for professors to delegate review work to their graduate students. Is this ethical?

    There is a distinction between reviewing for conference submission (in the sense of STOC/FOCS) and refereeing a final version for a journal. In the latter case the paper has almost always already appeared in some form or other. If it has appeared previously then having one's graduate student work on a referee report to check details is usually reasonable, though one should guide the student to the correct overall conclusion about the paper. (In fact, editors knowing the time commitment involved often explicitly ask for this.)

    On the other hand if one is doing STOC/FOCS reviewing then unless one is the PC member it is not OK to pass things off to a grad student. The whole point there is the level of judgement being asked of the reviewer that the student should not be expected to have.

    ReplyDelete
  10. In my experience the professor always says to the editor, for example, "I will review this, along with one of my students."

    Learning to review papers is an important part of the apprenticeship.

    ReplyDelete
  11. On the other hand if one is doing STOC/FOCS reviewing then unless one is the PC member it is not OK to pass things off to a grad student.

    Funny, what I've seen in my years in the field is almost the complete opposite. Journal papers tend to be yours alone to review, since you are explicitly given a single paper and lots of time to referee it. In contrast, for conference papers you are given a 30 to 50 papers to read and very little time to do so, so it is only natural to assume that third party help will be used.

    However as a PC you *must* read the reports from your subreferees and renormalize them according to your own scale.

    ReplyDelete
  12. So the question now is: why shouldn't submitted work be automatically public? Maybe if that was the default, people wouldn't rush to submit without working out their results properly... That is a serious problem in our community.

    ReplyDelete
  13. Clearly there is a problem when Janos' objection (to Scott) is taken to extremes: Imagine that the person decides they no longer want to publish their paper at all, or that they will get back to it in 5 years, when they have more time. Does this mean that (hypothetical-) Scott can never base any work on that paper? Can scientific or (even moreso) mathematical truth be "private"? (The answer is no.)

    ReplyDelete
  14. I agree with Anonymous above--I think all submitted papers should be public. Additionally, I think all reviewer comments should be public. That would help in the reviewing process. If you see a paper has been rejected for particular reasons and those reasons have not been addressed, then you can save the effort of re-refereeing the paper (if you agree with the reasons). It did happen to me once that I refereed a paper and found an error. The paper was rejected and the author was notified of the error. A couple months later, the exact same paper appeared in another top conference, error and all. Also, if all submitted work is public, then all submitted work will be treated equally and thus fairly.

    ReplyDelete
  15. Funny, what I've seen in my years in the field is almost the complete opposite. Journal papers tend to be yours alone to review, since you are explicitly given a single paper and lots of time to referee it. In contrast, for conference papers you are given a 30 to 50 papers to read and very little time to do so, so it is only natural to assume that third party help will be used.

    However as a PC you *must* read the reports from your subreferees and renormalize them according to your own scale.


    As a PC member you are in a role as editor so it is of course OK to ask a 3rd party. That 3rd party should have some expertise which is why I have the following to add to Lance's list:

    - It is unethical to give a conference review paper to a sub-referee in order to educate the sub-referee.

    (If one is not a PC member, it is unethical to pass on a paper given to you as a sub-referee to anyone else, but Lance already covered that.)

    For journal refereeing of a paper whose results have already appeared in unrefereed conference form, it is fair to use the process as part of educating one's graduate students.

    ReplyDelete
  16. And if a paper has been submitted to a journal, but has not been published in a conference proceedings, but will soon appear in a conference, before the 5th of the following month, and the author has given 3 talks (2 in the US) about said paper, but no graduate students were allowed to attend, then it is ok to use one's students? What if the author is himself a student, but soon will graduate, before completion of the review? Should he review his own paper? I'm confused.

    ReplyDelete
  17. - It is unethical to give a conference review paper to a sub-referee in order to educate the sub-referee.


    Educate as in "learn something about the subject" or educate as in "learn to referee"?

    ReplyDelete
  18. I agree with Lance's rules completely.

    What do you do when you mistakenly violate rule number 1 and substantially improve a result for a paper you are refereeing, but your improvement is so substantial, that you don't feel generous enough to share with the authors for an anonymous thank you? The logical thing to do if you follow rule #1 (if the original result is nice enough without your improvements) is to WANT THE PAPER to be ACCEPTED! Why? Because then when it's accepted you are free to publish your improvements! I've seen some shockingly slimy cases of people killing a paper because they've improved it and then tried to write up their improved results claiming that they never saw the original paper!

    In this situation, if you are ethical, and the original paper was nice to begin with, you hope the paper gets accepted as soon as possible, because then you can write up your improvements and cite the original.

    ReplyDelete
  19. The problem with using graduate students comes when it is hard to enforce that they follow rule #1. If the advisor is in a position to enforce rule #1, then it's good to have them learn by reviewing. I agree with the commenter who said that there's less risk with journal refereeing if the paper has already appeared in conference form, so the results are "out there". Of course the editor should be told the graduate student was involved as a routine matter.

    ReplyDelete
  20. I've seen some shockingly slimy cases of people killing a paper because they've improved it and then tried to write up their improved results claiming that they never saw the original paper!

    Sadly this is becoming more common.

    ReplyDelete