Thursday, April 15, 2021

Ordering Beauty

First, congratulations to fellow complexity theorist and blogger Scott Aaronson for receiving the 2020 ACM Prize in Computing for "groundbreaking contributions to quantum computing". The prize is ACM's highest honor for mid-career researchers. Well deserved! 

Now back to our regularly scheduled post...

Every freshman at Cornell back in 1981 had to take two seminar courses, basically one-shot courses in an usually humanities area which required no prerequisites but lots of writing. I took my first course in philosophy. The instructor, a PhD student, at one point described his research, a philosophical argument that there is an intrinsic total ordering of beauty, say that Beethoven would always sit above the Beatles, no matter the beholder. I didn't believe him then and still don't today. A few months ago the Washington Post ran a story with the same theme entitled Maradona was great, and maybe the greatest. Can we make similar claims about artists?

Somehow we have this belief when it comes to conference submissions, that there is some perfect ordering of the submissions and a good PC can suss it out. That's not really how it works. Let's say a conference has an accept rate of 30%. Typically 10% of the submissions are strong and will be accepted by any committee. About half the submissions are just okay or worse and would be rejected. The other 40% of the submissions will be chosen seemingly randomly based on the tastes of the specific members of the program committee. Experiments in the NeurIPS and ESA conferences have bourn this out. 

Why not make the randomness explicit instead of implicit? Have the PC divide the papers into three piles, definite accepts, definite rejects and the middle. Take the third group and randomly choose which ones to accept. It will create a more interesting program. Also randomness removes biases, randomness doesn't care about gender, race and nationality or whether the authors are at senior professors at MIT or first year grad students at Southern North Dakota. 

We put far too much weight on getting accepted into a conference given the implicit randomness of a PC. If we make the randomness explicit that would devalue that weight. We would have to judge researchers on the quality of their research instead of their luck in conferences.

Given that conferences, especially the virtual ones, have no real limits on the number of papers and talks, you might say why not just accept all the papers in the middle. Works for me.


  1. I agree wrt randomizing! It seems more fair and would make the job of the PC less arduous. A similar approach seems applicable to undergraduate admission to top colleges. Ivy League colleges, for instance, often say that they could fill several (essentially indistinguishable) classes in any given year. Why not have three groups -- accept; not accept; and lottery? And...let folks know when they made it to the lottery.

  2. Seriously, though...12:07 PM, April 15, 2021

    ... why not just accept that 40% as "poster-only" papers? Or, do a lottery with Pr[accept] = 1?

  3. If you do it at random there may still be some debates on the edges:

    Is this paper OBVIOULSY GOOD or should it go into the random pile?

    Is this paper OBVIOUSLY not-worthy or should it go into the random pile.

    This is not an objection, just an observation.

    Another thought- from what I have seen the number of papers that are worthy of FILL IN YOUR FAVORITE PRESIGE CONF has skyrocketed. Multipl sessions help, but Lance is right-- if its going to be online anyway
    than the usual constraints don't hold and we could let more papers in.
    How many? Thats a harder question if you want to maintain some notino of prestige conference (and maybe you don't).

  4. One new option to add toward better fairness, transparency, and competition is to publish the list of rejected papers with access links.

  5. Great idea, please try to make it happen. It is important that computer scientists take the lesson that randomness can be very powerful into the real world where it matters.

  6. Nice log entry. (+1) for mentioning Diego.
    Two concerns.
    [1] circularity.
    If we already assume there are three
    categories of entries, definite accept, definite reject,
    and randomized accept, then we already
    admitted that there is some absolute ordering
    [2] Randomization, however conducted, in the ultimate
    scheme will also always be biased -- aka biased on the
    pseudorandom number generators chosen.