Rather than inventing exercises, the authors extracted their tasks from the workflows of AI products already adopted by the market. The observation is stark.
Most agent test benches are based on tasks designed by the researchers. StartupBench, launched on August 18, 2026 by a team of 38 contributors led by Liya Zhu, takes the opposite path: the authors left AI products already validated by their market, studied their workflows and their users, and then derived end-to-end tasks covering several professions.
The result tempers the ambient enthusiasm on general agents. Even the most successful model only completes about 30% of the tasks, while it makes substantial partial progress on many of them. The analysis of failures points to two dominant causes: the monitoring of complex instructions and the specialized expertise specific to each field.
The authors' conclusion is clear: much of the labour flows that the market has already validated remain beyond the scope of the reliable capabilities of current generalist agents.
Read the full article on the original source
Go to the source →