By replaying the experiments of 168 papers presented at a machine learning conference, AI agents were only able to confirm a minority of the claimed results.
SAI Labs screened 168 papers selected for oral presentation at ICML 2026 with AI agents to extract the central claims from each text, download the associated material, replay the experiments when possible, and compare the results to those announced by the authors. Of the 92 papers with at least five verifiable claims, agents were able to reproduce more than two out of five claims for only 34 of them — and more than 80% of the claims for only eight papers.
The most striking episode concerns a theoretical chemist at the Zhejiang Lab in Hangzhou, who used an AI system to predict molecular boiling points: the model produced values that contradicted data considered to have been acquired for 75 years in a reference database. However, specialists point out that these automated verification tools remain unreliable arbiters of the scientific literature, even if they are proving effective in spotting flaws in old articles or databases.
Read the full article on the original source
Go to the source →