ZipBench compresses LLM benchmarks maintaining a correlation of 0.98 with the original

A method presented at EMNLP 2026 reduces the size of the evaluation sets by running only a small number of reference models, with mean absolute errors between 0.002 and 0.02.

Evaluating a language model on large test suites is expensive, and much of this cost is redundant: many items measure the same thing. This is the starting point of ZipBench, submitted on September 11, 2026 by Zhongzhan Huang, Junxin Li, Guoming Ling, Yupei Lin, Shanshan Zhong and Hefeng Wu, and retained in the main track of EMNLP 2026.

The method really only evaluates a small set of anchoring models, then synthesizes fictitious evaluation results to broaden coverage, learns compact representations of items and finally selects a small but representative subset. The whole remains inexpensive to produce, which was the main obstacle of previous compression approaches.

On the rankings, the compressed versions showed mean absolute errors of 0.002 to 0.02 and an average Spearman correlation of about 0.98 with the complete benchmarks. The authors publish a "ZipBench Zoo" of more than 100 compressed versions covering textual, multimodal and agentic tasks, making the evaluation accessible to teams without a significant calculation budget.