In light of rapid advances in generative AI and their impact on research and scientific communication, STOC 2027 is experimenting with several new policies intended to encourage high-quality submissions and promote clear and effective communication of research.
Let's talk about these changes, which seem more designed to limit the deluge of AI generated papers.
STOC 2027 submissions will not be anonymous; all listed authors must be human and are responsible for the submission.
This reverses the move to removing authors' names that STOC made in 2023. I was never a fan of double blind reviewing and you need authors who can take responsibility for the submission.
Each author may appear on at most five submissions.
Understood but it might hurt some students who have an active advisor.
Every paper must be submitted to arXiv before the STOC paper submission deadline. Authors must provide a public arXiv URL or proof of arXiv submission along with their submission PDF, which must be identical to the arXiv version.
In the past you could submit preliminary results and try to extend them before publication, since reviewers were expected not to build on unpublished work they were reviewing. This rule may cause some authors to hold back submissions or use AI to help with the extensions.
Authors must submit a video explaining the work, its context, and its innovations relative to prior work. The video should be 20–30 minutes long and will be due 1–2 weeks after the paper submission deadline. The recording should be presented by at least one listed author. The written submission remains the primary object of review.
This rule will both check that at least one author understands the paper and add some friction to just generating papers using a few prompts. AI could generate the video of an author explaining the paper, but at least for now that would be prohibitively expensive. Some might use AI to generate the script but that would likely be easy to tell from the video.
I worry that judging the paper based on the video will hurt those who aren't native speakers of English, and might exacerbate unconscious biases so I hope the reviewers really do focus on the paper for the actual review.
Authors may use large language models (LLMs) and other generative AI tools in preparing papers. Substantive use must be disclosed in the paper; minor copy-editing and grammar or clarity improvements to the authors’ own text do not require disclosure.
I would go further and make all papers have an AI disclosure, even if it is just minor copy-editing or "No AI was used in the production of this paper".
Program committee (PC) members and external reviewers (sub-reviewers) may use LLMs to assist with reviewing. Authors must explicitly consent as part of the submission process. All reviews and decisions remain the responsibility of the PC members and sub-reviewers.
I would require consent as condition of submission especially since AI models already have access to arXiv papers. Recent AI models have greatly improved their ability to check proofs if prompted correctly so the overworked PC members can focus on paper quality.
We really need a larger conversation about the role of conferences in theoretical computer science if one can now generate papers from prompts. I've long argued that we should focus the conference more on connecting the community than the "journal that meets at a hotel". The STOC TheoryFest has helped but it would be great to get further away from lauding papers that have complicated proofs and focus more on the ideas that truly drive our field.
The wording is ambiguous as is, but I confirmed with Shachar (PC chair) that it is indeed as you were suggesting: authors are required to give consent as a precondition for submission. That is, you have a choice to submit or not submit, but if you submit, you must check the box that says you consent to LLM use in reviewing.
ReplyDeleteI think this part in particular is a bad policy. I write for people, not LLMs, and I put effort into my writing to make it readable (though I admit I may not always succeed!). I think if authors use LLMs they should have to consent to LLM reviews, and when authors didn't use LLMs they should have the option to give consent or not for LLM reviews.
If the PC is worried about handling the increase in # of submissions - as they should be - maybe they should set the paper-per-person cap at something less than 5, and require authors to agree to be reviewers as well (which more or less automatically scales; to make it work you might need to do something like scale appropriately by number of submissions & number of coauthors, but there are presumably many variations that could work).
Allegedly LLM-less authors should still consent to an "Is this AI" check (which in itself uses an LLM).
Delete> I would go further and make all papers have an AI disclosure
ReplyDeletestark contrast to my opinion being that I'm tired sick of Ai being mentioned all the time, so if it wasn't used in disallowable quantities - just let it not be named at all
AI disclosures will soon be like disclosing that you used a calculator. We'll just assume people are using it.
DeleteI hope not, but also I don't think so. People can use it in many different ways, and I think it's pretty crucial that they are transparent about that. Some people (like me) use it not at all. Others use it only for minor things like editing grammar and spellcheck. Others use it to check for conceptual/logical errors, or to help find citations. Others use it to find proofs for them - some for small lemmas, some for the main results of the paper. Others use it to write the whole paper and the human only does minor editing after. Those are *very different*, and authors deserve different amounts of credit in those different situations. Thankfully, with the exception of your one comment (Bill), most mathematicians & computer scientists I talk to - in the full range of extremely pro-AI or to extremely anti-AI - seem to be on board with transparency about its use.
DeleteThe assumption that "everyone does it" is always what bad people assume to justify their own degenerate behavior to themselves. The truth is lots of people don't use any form of Gen "AI" whether for ethical, environmental, intellectual, or other reasons.
DeleteBill, you're right that people without virtue who don't fundamentally value any of these aspects over the potential prestige benefits of prompting their way to generating resolutions of open problems in this interim period where the community still isn't sure what sorts of mathematical behavior they should incentivize and encourage will assume everyone uses it.
You're wrong that everyone will make this assumption.
There will still be plenty of people who don't.
I like the IDEA of the submitters submitting videos. And this may be part of the more general transition from rewarding COMING UP with (say) theorems to DIGESTING and COMMUNICATING what our AI-overlords tell us is true. But reviewing all videos of papers that look promising will be an gigantic task. Maybe we can't do it. Maybe we'll have AI do that also.
ReplyDeleteIt is interesting and we should adopt our processes and structures to the changes.
ReplyDeleteI wonder though about the implications of changes, e.g. we have had great mathematicians who were terrible at explaining things simply.
AI is much better at explaining vs. discovering, when compared to many top human researchers.
What people want out of a coffee varies, and I understand more senior people with tenure are not focused on advancing science, but more junior people are looking for establishing reputation in the field and they are unlikely to come up with ground breaking new conceptual ideas (Turning did, of course, has it become much more difficult as our girls martures? how many PhD students or junior faculty members have had original ground breaking ideas over the past 30 years?).
For junior people publishing papers is like internship for medical science. They are trying to demonstrate their skills and potential and maybe even independence.
How AI will impact the researcher career paths, from PhD to postdoc to junior faculty to senior faculty?
A lot of interesting challenges to figure out.
The PC chair is, as we speak, preparing a blog post explaining these changes (and the motivation behind them, as discussed by the PC). It may be worth waiting for that to have this discussion.
ReplyDeleteI am wondering what about privacy where authors do not want to reveal their faces or voices or identity?
ReplyDeletePaper bag over their head at the conference?
DeleteI was investigating a tcs problem using AI for a few days, guiding it with instructions on how to solve the problem and make experiments, but I couldn't get it or me to find an answer. Then, I just gave it the broader theorem that I'd like to show (it's a problem that's been open for more than 10 years) and it gave me a proof in a few minutes. It asked me to download a few papers that were not accessible to robots and made small corrections. I asked it to write a paper that I'm currently verifying without AI. It is not easy to verify, because it uses many techniques that I'm not familiar with and I have to read at least 3 other papers to be able to assert anything with confidence. I get the feeling that it's not my paper, as the ideas in it are not mine at all and I wouldn't be able to answer deep questions about it. At the same time, I think it deserves to be published in a good conference, because the result and the way it combines techniques is important. I'll probably end up putting my name on it and sending it to a good conference with a disclaimer about AI, but I feel like I'm cheating.
ReplyDelete