The new King and Worthington working paper, AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices, is interesting.
Not so sure about Kent Anderson’s comments on it, especially where he says that peer review is “first and foremost a selection step”, given that in 2022, mega-journals, typically associated with scientifically sound rather than important and novel work, published more than 300 000 articles, almost a quarter of the biomedical literature that year.
Plus King and Worthington explicitly say their study is testing constructive feedback intended to improve a manuscript during peer review, which I think is widely regarded as one of peer review’s core functions, alongside helping editors make publication decisions.
What they have found is quite interesting; they are basically saying AI-generated constructive manuscript feedback outperforms an “unusually high quality set of [well-paid] human review” under their evaluation framework, with human editors preferring the AI 87% of the time in a sample.
There are some other interesting stats in the paper, such as 5.5% incorrect claims in the AI reviews versus 15.9% in human reviews.
One of the journal policy choices they suggest in their conclusion is:
“some journals may begin to recommend or require that authors conduct repeated AI reviews themselves before submission, iterating with AI to improve their papers (via what are essentially self-supervised revise-and-resubmits), and continuing until authors are convinced they have resolved all resolvable issues and improved the paper as much as possible within time or resource constraints”
I think this is already happening formally and informally to some extent. Taylor & Francis, for example, adopted Paperpal Preflight in 2024 as part of its editorial workflow.
Whilst I largely agree with Kent that the vast volumes of thin, sloppy, commercially driven or downright fraudulent science aren’t doing much for science as a whole, this problem is only going to get worse. Unless the global research community can somehow come together and instantly change everything from career incentives and research assessment, institutional performance metrics and funding systems to publishers’ business models, the volume problem is not going away. The community needs to find ways to deal with this, and using AI is one possible approach.
