Evaluating AI agents is expensive: keeping only the tasks of intermediate difficulty is enough

Across 8 benchmarks, 33 agent scaffolds, and more than 70 configurations, filtering out tasks with a historical success rate between 30 and 70 percent eliminates 44 to 70 percent of the work.

Evaluating an agent on a comprehensive benchmark costs thousands of dollars in model calls, discouraging systematic comparisons. Ndzomga questioned whether a reduced subset of tasks was enough to preserve agent rankings.

The protocol covers eight benchmarks, thirty-three agent scaffolds, and more than seventy model configurations. The central finding is that rank prediction remains stable even as absolute scores move from one evaluation framework to another — a behavior that distinguishes agent benchmarks from static language model benchmarks.

The proposed rule is frugal: keep only tasks with a historical success rate between 30 and 70%. This filtering by intermediate difficulty removes 44 to 70% of the tasks while preserving the fidelity of the ranking, and does better than random sampling. Conversely, a gluttonous selection of the most discriminating tasks deteriorates as soon as the distribution changes.