Do personal agents really progress through experience? PAST-Bench separates gain from its mechanism

Over 26 scenarios and 204 episodes, the benchmark turns memory on and off to verify that the progress observed is going through the intended channels — which is not always the case.

Posted on August 4, 2026 on arXiv by Shuhan Xue, Zixin Ding, Yichen Shen and their colleagues, PAST-Bench tackles a question rarely asked in front of it: when a personal agent seems to improve with use, does the improvement really come from the experience he has retained?

The protocol consists of running the same tasks with and without experience, under controlled conditions, and instrumenting the three tracks that are supposed to produce the gain — saving, finding, updating. The bench covers 26 scenarios and 204 episodes spread over four capacity families: memory, procedure reuse, information gathering and updating mechanisms. Seven basic models and four cadtriciels of agents were screened.

The central result is a discrepancy. Progress is real but very uneven according to abilities, and above all identical overall scores can cover very different situations: there is no guarantee that the gain will take the mechanism envisaged. Hermes+, an enriched variant tested by the authors, notes the average gain from experience and provides clearer indices of correct activation of the mechanisms — but the benefit remains dependent on the model and capacity considered. The code is published on GitHub.