I hate that we have to judge theorists but departments need to make decisions about PhD admission, faculty hiring, tenure and promotion. Artificial intelligence adds new wrinkles to the process. AI makes it easier to prove theorems and write up papers, certainly leading to more papers. And how do we judge the role that an author plays in an AI-assisted or AI-generated paper? And are we rewarding good prompting and access to the pricier AI models?
These issues go back before the AI age. As our field has become more metric driven, we've seen a significant increase in the number of research papers from undergrads, grad students and young professors, often at the least publishable unit, just good enough to get accepted at various conferences. Bill told me about a quote he heard, "Deans can't read but they can count." Too often even our CS colleagues will count papers as well.
When I was department chair, I had faculty candidates upload a single paper with their submission. That gave us a single paper to read, though I'm not sure it always got read.
Since our papers list authors alphabetically instead of by contribution, we already had challenges sussing out the role an individual author played in the paper. With AI papers we often get an AI disclosure, but once we use that for judging a theorist, people will write those disclosures carefully, maybe in a misleading way or worse.
Here we could rely on recommendation letters. In computer science, letter writers tend to write the most positive letter without lying, so to read a letter properly you need to reverse that process. But now that many of these letters seem to be AI generated themselves, that muddles the information we get.
How do real companies deal with hiring? A company might wait until it gets a better feel for just what roles AI will play in research and teaching, but academic departments tend to fill slots because they may not have them next year.
Many companies use AI to at least give a first pass through the candidates, some even doing the first interview. We could do the same. AI would certainly do a more thorough job reading candidates' statements, papers and recommendation letters than we do, especially for those applying from universities we know less about. An AI interview could check how well a candidate understands their research and the role they played. But I suspect CS departments would feel queasy about offloading these processes.
A group of theory faculty, responding to a Simons Institute working group report, put out a statement about hiring for the AI age. You could do worse than following these guidelines.
- Emphasize quality over quantity. In the evaluations of candidates, consider at most three works from July 2026 onwards (of the candidates’ choosing), in addition to any number of prior works before July 2026.
- Actively assess thorough understanding. Evaluate the thoroughness of the candidates’ understanding of the aforementioned works more than was done normally until now. This evaluation could take the form of a “chalk talk,” more mathematically substantial interview slots, and/or other methods of the department’s choosing.
Why did it take the AI era for us to finally put quality over quantity?
No comments:
Post a Comment