OpenConjecture: Out of 200 random guesses, GPT models offer a 60% solution, but not always the one expected

Rather than judging AI on its most publicized breakthroughs, researchers tested it on a random sample of recent conjectures. Trivial counter-examples and unoriginal evidence are alongside real novelties.

For the past year, announcements of conjectures resolved by language models have multiplied. But these successes are sorted out on the track and do not say anything about the failure rate or the calculation spent. Helen Jenne, Davis Brown, Jesse He and Henry Kvinge thus constituted OpenConjecture, a set of conjectures extracted from mathematical articles published on arXiv since February 2026, and then submitted 200 of them, randomly drawn, to advanced GPT models, in three passes, according to a reproducible protocol and without any mathematical indication specific to each problem.

More counter-examples than evidence

The models put forward a candidate solution for about 60% of the conjectures, and counter-examples are approximately 1.3 times more frequent than demonstrations. Without having each response checked by experts, the authors analyse their nature: some exploit an incomplete error or formulation of the statement, such as proposing the trivial group as a counter-example, down to the intent of the authors; several evidences are derived fairly directly from results already published; However, others have at least moderate novelty.

The study argues for assessments that do not merely verify that a response is correct, but also examine its fidelity to the statement, how it cites existing literature and the reality of its mathematical contribution.