Overview
The Association for the Advancement of Artificial Intelligence ran an AI review pilot for the main track of its AAAI-26 conference, in which every paper in the first review phase received one clearly labelled AI-generated review alongside two human reviews. The AI review gave no scores or recommendations and did not replace any human reviewer. Reviewers could see it after submitting their own reviews, as could senior programme committee members and area chairs.
A multi-stage system combining frontier models, tool use and safeguards generated reviews for all 22,977 papers in less than a day. OpenAI provided the API credits for the pilot as a conference sponsor. When it announced the pilot in May 2025, AAAI said LLMs would also help senior programme committee members by summarising reviewer discussions and showing where reviewers agreed and disagreed.
Benefits and outcomes
- In the organisers’ survey of 5,834 authors and programme committee members, 53.9% found the AI reviews useful and 20.2% did not.
- Survey respondents rated the AI reviews higher than human reviews on six of nine criteria, with the largest differences for identifying technical errors and raising points not previously considered.
- Survey respondents rated the AI reviews worse than human reviews for overemphasising minor issues, containing technical errors and including wrong or unhelpful suggestions.
- On the organisers’ SPECS benchmark for detecting scientific weaknesses, the system’s recall was 0.64, compared with 0.43 for a baseline LLM review.
Notes and quotes
- AAAI President Stephen Smith described the pilot as “a careful, measured approach to incorporating new technology into the scientific review process”.
- The authors of the evaluation say self-selection bias must be considered when interpreting the survey results.
Sources
- AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot, Joydeep Biswas and others, arXiv, 15 April 2026
- AAAI Launches AI-Powered Peer Review Assessment System, AAAI, 16 May 2025
- AAAI-26 Main Technical Track Call, AAAI, 9 June 2025
- The AAAI 2026 AI-Assisted Peer Review Pilot Program, AAAI, January 2026
This entry was written with the help of AI, from the sources listed. Please check the sources before relying on it, and let us know if anything needs correcting.
