14,767 evaluation papers scrutinised: what the benchmarks reveal about expectations for LLMs

A mapping exercise covering the period from January 2022 to August 2026 reveals a shift in benchmarks towards action, interaction and professional roles — and an increasing role for models in their own evaluation.

Chao Wang, an independent researcher, submitted a study on 15 September 2026 which took the benchmarks themselves as its object. Through progressive filtering and automated coding of the full text, it analyses 14,767 articles introducing or updating an evaluation resource, published between January 2022 and August 2026, to reconstruct what the community really expects from language models.

The resulting picture shows a shift towards action, interaction and professional usage, far from the only questions and answers of the beginnings. LLM notation is widespread in evaluation protocols, whether or not they involve agents, while the production of test equipment by the models themselves has not really progressed in recent times. Old and new ways of designing a benchmark coexist without the former disappearing.

The author points to the fundamental difficulty: when models are involved in both the construction of tests, their resolution and the scoring of responses, the assessment may reflect their own preferences and blind angles rather than constitute evidence independent of their abilities. The article is 15 pages long, with five figures and seven tables; data and code are published.