AgentWorld: after more than fifty rounds of cooperation, the best model only succeeds in one out of every two tasks

A test bench places between 3 and 20 agents in asymmetric roles within an MMORPG-style sandbox. None of the models tested achieved a success rate of more than 52 per cent, and a new metric measures the effort that is actually useful.

Presented on 25 September and accepted at the COLM 2026 Conference, AgentWorld tackles a blind corner of agent evaluations: the cooperation that lasts. The test bench has 100 human annotated tasks, plus 100 augmented variants, played in an environment inspired by MMORPGs where 3 to 20 agents with intentionally unequal roles and abilities must coordinate over fifty interaction rounds.

The authors — Raphael Shu, Yusen Zhang, Young Min Cho and their colleagues — also introduce a metric, the Causal Collaboration Effectiveness. Rather than counting successes, it reconstructs the graph of causal dependencies between actions and estimates what fraction of the collective effort actually contributed to the result.

The scores are severe. On Gemini 3 Flash, Claude Haiku 4.5, GPT-5 Mini and DeepSeek R1-70B, the best model capped 52% of successful tasks. Three failures come back: exchanges that inform no one, confusion about who does what, and inability to maintain a shared plan from one turn to another. The test bench shall be published in open access.