Refereeing: Go with your Guts, or Let Reason Rule?

My usual way to review conference submissions is to ask myself a few standard questions:

1. What is the problem being studied? Is it motivated in a compelling manner, by either practical or theoretical considerations? New problems warrant more careful scrutiny.
2. What is the result? If there are several results, usually I only look at the main result and evaluate the paper based solely on its best result.
3. Is the proof clever or difficult? Can I identify the key idea, and is it novel?
4. Finally, I give the paper a bonus if it is particularly well-written and a big malus if it is particularly poorly written, especially if the authors are all well beyond their student years - I get impatient with them, and think to myself that they should know better.

At this point in my career, this is usually all fairly routine. However, as Daniel Dennett suggests: "Perhaps our approximation of a perfect Kantian faculty of practical reason falls so far short that our proud self-identification as moral agents is a delusion of grandeur. " I was recently given a paper to review for a conference, and something strange happened. Based on my usual criteria outlined above, I sent to the program committee a mild recommendation for rejection and promptly put the submission in the trash. But instead of instantly forgetting about it, I kept remembering bits and pieces of it and found myself trying to reconstruct parts of the proofs. After a couple of days, it dawned on me that, even though the submission did not pass the filter of my "objective" criteria, still, I was interested in it and actually liked it!

I wonder what one should do in that case. Should you trust your instinct, or obey your evaluation rules? Go with your Guts, or Let Reason Rule?

Comments page (Usual comments link does not work properly, use this instead.)

Claire