Since there is now a CCC Test-of-Time Award, see here, (CCC stands for Computational Complexity Conference), I decided to look at other Test-of-Time awards in computer science.
Below is a list of various computer science Test-of-Time awards, along with their
eligibility requirements.
1) IEEE Visualization (VIS) Test of Time Award, see here. VIS=Visualization. Their 2023 award page says:
Papers are selected for each of the three historic conferences (VAST, InfoVis, and SciVis). ...
This year VAST gave out a 10-year test of time award, InfoVis a 10- and 20-year award, and SciVis a 13, 14 and 25 year award.
2) SC Test of Time Award, see here. SC=Supercomputing Conference. Their site says:
Papers appearing in the SC Program 10 to 25 years prior are eligible.
3) CIKM Test of Time Award, see here. CIKM=Conf. on Information and Knowledge Management. Their page states:
The Test of Time Award is given for CIKM papers published at least ten years ago.
3) ACM SIGCSE Test of Time Award, see here. SIGCSE = Special Interest Group on Computer Science Education. ACM=Association for Computing Machinery. The name is from the era when computers looked like angry filing cabinets. The ACM SIGCSE Test-of-Time page states:
The award recognizes an outstanding paper published in the SIGCSE community.
Note that this is broader than just their conference. Good!
4) LICS Test of Time Award, see here. LICS=Logic in Computer Science. Their award page says:
The LICS Test-of-Time Award recognizes a small number of papers from the LICS proceedings from 20 years prior.
5) STOC Test of Time Award, see here. STOC = Symposium on Theory of Computing. Their page states:
There are three awards---papers presented at the STOC conferences in 10, 20, or 30 years ago [they later say there is some flexibility on that].
6) FOCS Test of Time Award, see here. FOCS stands for Foundations of Computer Science. Their page states:
The awards recognize papers published in the Proceedings of the Annual IEEE Symposium on FOCS. [From elsewhere on the page they have three categories: papers 10-years ago, 20-years-ago, and 30-years ago, but they have some flexibility with that.]
7) SIGecom Test of Time Award, see here. SIGecom = Special Interest Group---Economics and Computation. Their page states:
...must have been first published in an archival journal or conference proceedings 10–25 years before. At least one author must not be dead. [A surprisingly nontrivial constraint in a Test-of-Time award.]
I think this is the only award on this page that stated the not-dead criteria explicitly. Are people in Economics and Computation publishing papers into their 90's? While in hospice care?
8) NeurIPS Test of Time Award, see here. NeurIPS = Neural Information Processing Systems. I couldn’t find an eligibility description, so I asked ChatGPT, which confidently stated:
Yes—the paper must have been published in NeurIPS at least 10 years ago.
If this is wrong, please let me know. ChatGPT is sometimes confident the way undergraduates are confident before seeing the homework solution.
9) CCC Test of Time Award, see here. CCC=Computational Complexity Conference. Their page states:
Eligible papers are those that appeared in CCC (or Structures, pre-1996) at least 10 years ago.
------------------------------------------------------------------------
These awards raise some questions:
1) Why tie eligibility to a conference? Suppose Abisola proves P ≠ NP and publishes it in Annals of Mathematics. Under many of these rules, she would be ineligible for most theory Test-of-Time awards.
CON: Important results may not appear in the “right” conference or any conference at all.
PRO: None. Absolutely none. Why is it done? To publicize the conference and provide a nice ceremony venue. These are not strong intellectual reasons. (I had a nastier comment about this but ChatGPT convinced me to be more polite.)
My proofreader brought up the following:
a) Who will give out the award? Answer: An organization like the ACM, and it DOES NOT need to go to an ACM conference.
b) How to limit what papers are eligible? Any paper that was in a conference or journal (though see item 3 below). That seems like a rather large set of papers to consider; however, after 10 years we DO know which papers stood the test of time. As an example- when Lance made his best theorems of the decade lists he did not pay attention to which venue the result appeared in.
2) Why the 10–25 years ago window? I understand waiting 10 years—time needs time. But why exclude older papers? What if the paper Some Connections Between Bounded Query Classes and Nonuniform Complexity (STRUCTURES/CCC 1990) suddenly becomes important? Too bad: it aged out, like a carton of milk.
Are there examples of papers that became important many years after they were published?
I have one sort-of example: Google AI Overview says
The permanent of a matrix was first introduced independently by Cauchy in 1812 and later independently rediscovered by Binet in 1812. [The identical years makes me suspect, and also makes the notion of ``later'' rather odd- Did Cauchy do it in February and Binet in June?]
The permanent became much more important after Valiant, in 1979, showed that it was Sharp-P-Complete. So perhaps Cauchy's paper should get a test-of-time award.
Fun Fact: The Wikipedia entry on The Permanent (see here) does not mention Valiant, though there is a separate Wikipedia entry on Computing the Permanent (see here) that does.
3) Why require publication in a journal or conference at all? Suppose Perelman proves P ≠ NP, posts it on arXiv, and never submits it anywhere. Perelman did just that with his proof of the Poincare Conjecture, and that was good enough for a Fields Medal.
This touches on the much broader---and increasingly relevant—question: What is the future role of journals and conferences in the age of arXiv?
4) (Bonus question): Is there any real difference between the STOC conference and the FOCS conference in terms of the scope of the papers? Was there ever? I would guess no and no. Maybe some of my older readers can tell me, unless they are too busy writing papers in Economics and Computation.
5) Another proofreader pointed out that it would be a good idea to live up to the title of this post, Test of Time Awards: A Good Idea but, and say why they are a good idea instead of bitching and moaning about eligibility. Good Idea! Some thoughts:
a) They might encourage researchrs to aim for deep contributions, not just fashionable ones. I doubt this is true since I doubt authors think about which award they might win.
b) They reward long-time impact over short-term excitement. Is it much of a reward? How do they benefit the recipient? I ask non-rhetorically.
c) They are an objective record of subjective opinion. Useful for historians!
We give Best Paper awards to papers in conferences in a given year, but there are fashions of the time that influence these awards. The Test of Time awards are a re-balancing of this judgement. Sometimes the original judgement passes the test of time, but acknowledging work that was originally overlooked makes these awards very interesting.
ReplyDeleteThere is a big advantage for these awards of bounding the number of eligible papers in some way even within a specific conference. Some awards specify the precise year, some give a bit of leeway (for example STOC/FOCS give priority to a given year of their conferences but allow consideration of papers shortly prior to those years; others like LICS have a 3-year window).
The awkward fact for many awards is that though they may have a formal procedure for nominations, there is a lot of randomness in who gets nominated. In particular, for Test of Time Awards there are often more deserving candidate papers than those that have been nominated.
This is why it is good that for most such awards, the procedure is set up so the committee can choose other awardees than merely those nominated. In particular, for the FOCS/STOC Awards, the committee focuses on the papers from the specified years and makes sure that they have properly considered all the potential candidates from those years. I expect that a majority of all awardees are chosen in this way.
I wonder how the new CCC Test of Time Award is going to work in this regard. As usual, the committee gets to choose a paper that was not nominated. Though CCC has many fewer papers than STOC/FOCS per year, there are 30 years of papers that will be eligible. The job of choosing just 1 paper from such a large number seems tricky. There will be many deserving candidates and the window that needs to be considered will only grow over time.
you said that there is a big adv for bounding the number of eligible papers (e.g., only STOC papers). What is that advantage?
DeleteI thought the answer was implicit in the rest of my comment ... Because of the randomness of nominations it is important for the committee itself to be able to weight all options carefully in the new light of what has gone on since the papers were published. For STOC/FOCS papers, the big influence has sometimes been outside of the collective expertise of the committee members, or been sufficiently tangential that they only have a rough sense that they have might need backing up. That forces the committee members to do research on individual papers. If the number of papers that they should consider carefully, and do this research on is too large, then that task becomes hard to do and they may resort to salience bias. Maybe that won't be an issue with CCC, but I wouldn't bet on it.
DeleteI didn't mention a separate reason that makes things hard if you let the time scale vary by a lot. It is much harder to directly compare the relative importance and influence of individual papers that are decades apart. It can be a tougher apples to oranges comparison than usual and rough quantitative measures like citations won't help much.
Why should "STOC test of time" award be given to a paper not at STOC? I honestly don't get most of the objections in the post. There are plenty of prizes that are given for strong work in any venue, why can't conference X celebrate strong papers that appeared at conference X?
ReplyDeletewe should be honoring excellent papers independent of where they appear. As for `plenty of pries for strong work'- Godel Prize, Turing Award, are there others? If so then you have a strong point.
Delete> Suppose Absila proves P ≠ NP
ReplyDeleteWhat is "Absila"?
We usually talk about Alice and Bob. When the show Bob Loves Abisohla came on the air I thought I would switch to using those names. Then I found out that Abishola is not really a Nigerian name, but Abisola is (I mispelled it in the post- but its fixed now). So think of Abisola as a generic name like Alice, Bob, Carol, Donna, Edgar, etc.
ReplyDelete