Overview
The Black Spatula Project is an open initiative testing whether large language models can find errors in scientific papers, how serious those errors are, and which models, prompts and pipelines work best. It looks for errors in maths, methodology, logic, figures and tables, and citations. Volunteers run papers, including withdrawn arXiv papers, through models such as ChatGPT, Gemini and Claude, and record their prompts and results in shared spreadsheets. Domain experts volunteer to check the results.
Anyone can submit a paper through the project’s GitHub page, and the analysis is posted as a comment within a few minutes. The project was founded by Steve Newman and is named after a paper about black plastic cooking utensils that contained a calculation error. In March 2025 it had eight active developers and hundreds of volunteer advisers, had analysed about 500 papers and was contacting authors about the errors it found.
Notes and quotes
- In March 2025, about 10% of the errors the system reported were false alarms.
- Each reported error has to be checked by experts in the subject, and Steve Newman said finding them was the project’s biggest bottleneck.
- In March 2025 the group was still testing which approach gave the best results.
Sources
- The Black Spatula Project
- The Black Spatula Project on GitHub
- No more calculation errors? AI checks papers for errors – Why that’s not enough, heise online, March 2025
- Oopsie prevention: AI tools now scan scientific papers to catch critical research errors, CO/AI
This entry was written with the help of AI, from the sources listed. Please check the sources before relying on it, and let us know if anything needs correcting.
