Agentic reproduction of social science results

Companies: ETH Zurich, Institute for Replication, TU Wien and 1 moreBuilt by: research, non-profit or individualStatus: Not statedYear: 2026

Overview

Benjamin Kohler, David Zollikofer and Elliott Ash of ETH Zurich, with Johanna Einsiedler of the University of Basel and Alexander Miserlis Hoyle of TU Wien, built a system in which AI agents reproduce the results of social science papers from a structured description of their methods and the raw data, without seeing the original code, results or paper. The reproduced outputs are compared with the published results cell by cell, and a language model traces each discrepancy back to its cause.

They tested four agent setups and four language models on 48 papers from 10 economics and political science journals that the Institute for Replication had classed as fully reproducible, and presented the work at EMNLP 2026.

Benefits and outcomes

  • The authors report that the agents could largely recover the published results, but performance varied a great deal across models, setups and papers.
  • The authors report that with the best setup, OpenCode with GPT-5.4, about 91% of reproduced coefficients had the same sign as the originals and more than 80% fell within the original 95% confidence interval.
  • The authors report that some results were not reproduced because the agents made mistakes, and others because the papers left out details of how the analysis was done.

Sources

This entry was written with the help of AI, from the sources listed. Please check the sources before relying on it, and let us know if anything needs correcting.