Staff are able to say where they are at the start and end of a task, but much less so in the middle

A test bench sets checkpoints throughout the execution: the reliability of progress reports plummets as the process goes on, which undermines the orchestrators who rely on them.

Agent architectures are often based on an implicit assumption: when a model announces where it is in a task, it can be believed, and trigger the rest of the flow accordingly. Boyang Wang, Yunhan Wang and Yalun Wu wanted to measure the robustness of this hypothesis. Their study, submitted on September 8, introduces StageIF, a controlled testbed in which checkpoints are distributed throughout the life cycle of a task, supplemented by an evaluation on the τ²-bench benchmark.

The result is a hollow curve. Most of the models tested maintain good accuracy in the early and later stages, but lose significantly in the middle of the journey — exactly where an orchestrator needs to know whether to wait, relaunch, or fail over. Newer models have a different profile, becoming more cautious as completion approaches.

The authors draw a direct recommendation for engineering: agent managers should not steer the progress of a task solely on the basis of the status reports produced by the model. It is less a question of a lack of capacity than of a gap between what a model can do and what it can say about what it is doing.