AI4AI-Bench: When asked to rewrite their own training algorithms, the score peaks at 0.25

A benchmark submitted on August 20, 2026 puts 29 agent configurations against ten families of training algorithms, with four hours of GPU per job. Recursive self-improvement is still very far from the mark.

The promise of AI that improves AI remains largely unverified due to the lack of a measurement instrument. AI4AI-Bench, released on arXiv on August 20, 2026, tackles precisely this hole: rather than evaluating data collection or hyperparameter tuning, it isolates an agent's ability to design a better training algorithm.

The protocol is based on ten frozen research repositories, covering ten families of algorithms. For each task, the agent has four hours on a B300 GPU to rewrite the algorithm. Twenty-nine configurations from six systems were passed on a scale, on a scale where 0 corresponds to a model with no input and 1.0 to the achievable optimum.

The average score is 0.166, the best system is 0.250. Two regularities stand out: the submissions that actually affect the learning procedure obtain 0.226 on average compared to 0.126 for the others, and extending the reasoning budget increases the average from 0.094 to 0.196. In other words, there is room for improvement, but the level achieved remains out of all proportion to the recursive runaway scenarios often mentioned.