Tuesday, August 15, 2006

Refereeing: Go with your Guts, or Let Reason Rule?

My usual way to review conference submissions is to ask myself a few standard questions:
  1. What is the problem being studied? Is it motivated in a compelling manner, by either practical or theoretical considerations? New problems warrant more careful scrutiny.
  2. What is the result? If there are several results, usually I only look at the main result and evaluate the paper based solely on its best result.
  3. Is the proof clever or difficult? Can I identify the key idea, and is it novel?
  4. Finally, I give the paper a bonus if it is particularly well-written and a big malus if it is particularly poorly written, especially if the authors are all well beyond their student years - I get impatient with them, and think to myself that they should know better.

At this point in my career, this is usually all fairly routine. However, as Daniel Dennett suggests: "Perhaps our approximation of a perfect Kantian faculty of practical reason falls so far short that our proud self-identification as moral agents is a delusion of grandeur. " I was recently given a paper to review for a conference, and something strange happened. Based on my usual criteria outlined above, I sent to the program committee a mild recommendation for rejection and promptly put the submission in the trash. But instead of instantly forgetting about it, I kept remembering bits and pieces of it and found myself trying to reconstruct parts of the proofs. After a couple of days, it dawned on me that, even though the submission did not pass the filter of my "objective" criteria, still, I was interested in it and actually liked it!

I wonder what one should do in that case. Should you trust your instinct, or obey your evaluation rules? Go with your Guts, or Let Reason Rule?

Comments page (Usual comments link does not work properly, use this instead.)

Claire

2 comments:

  1. If you can articular the REASONS why you
    like it then YES you can articulate them
    and argue for it.

    If you JUST like it in your gut but
    cannot articular why, then do not
    argue for it- you can't argue for it
    since you can't articulate.

    Do not be afraid of taking an unusual
    or odd stand- thats why program committees
    have lots of prople on them.

    WILLIAM GASARCH
    (Last time I posted a comment later
    comment refered to my comment as
    `anonymous said...' which was odd since
    I definitly used my name. THIS time refer
    to me by name please.)

    ReplyDelete
  2. Salut Claire ;)

    I received some comments from referee which exposed both "reasonable" and "guts" arguments: the referee said that he *wanted* to accept the paper because he felt that this direction should be encouraged, and then proceeded to point out the bad points of the paper (basically, bad writing in this case).

    As a referee, I don't see any problem in giving both opinions (reason ans guts): the PC member can pick from it. As a PC member, I guess that I would expose both arguments but vote with my guts: after all, the readers of the proceeding might have similar guts ;)

    ReplyDelete