The Replication Engine is an agentic system designed to automate parts of the process of checking whether the quantitative results reported in scientific papers can be reproduced from the authors’ code and data. Given a paper and its replication package, agents attempt to reconstruct the computational environment, run the analysis, connect regenerated outputs to the results reported in tables and figures, identify discrepancies between the paper and the underlying code, and suggest additional robustness checks.
This is an Institute for Replication (I4R) project led by economist Abel Brodeur, chair of I4R and associate professor at the University of Ottawa, and computer scientist Bruno Barbarioli, an I4R research scientist and University of Chicago PhD student. Their argument is that traditional replication is valuable but extremely labour-intensive, meaning only a tiny fraction of published research is ever checked. Recent advances in language models and coding agents potentially make it possible to perform some of these checks routinely and at much greater scale.
I4R has built and benchmarked an actual Replication Engine, initially concentrating on economics and political science. A 2026 evaluation used 74 papers and two frontier models. The agents agreed with reference assessments of whether replication packages executed about 99% of the time. Once the system had correctly linked a result in the paper to the corresponding regenerated value, it judged whether the values matched about 98% of the time. Detecting subtler differences between the methods described in a manuscript and what the code actually does remains much harder and still requires expert review.
A much larger version of the idea was published by the Institute for Progress (IFP) in August 2025 as part of its Launch Sequence, a collection of projects it wants people and institutions to build. The proposal envisages automated verification becoming part of the scientific publishing infrastructure. Papers would be checked at submission, receive green/amber/red indicators for different parts of their computational analysis, and eventually feed into knowledge graphs tracking the reliability and provenance of scientific claims. The proposed programme would initially cost about $10 million over three years and ultimately extend across fields including economics, psychology, computer science, physics and climate science.
Publisher/company: Institute for Replication (I4R)
Date of mention: 19/08/2026
Source: https://ifp.org/the-replication-engine/
GitHub: https://github.com/i4replication