A four-part assessment shows that state-of-the-art large models understand instructions very well, but remain significantly less reliable when it comes to physically handling objects.
A team of eleven researchers subjected four boundary models to a battery of robotic tests on the RoboColiseum platform. To put them on an equal footing, they developed Agent2Act, a control harness that provides everyone with the same observations, the same means of executing actions and the same feedback. Four components were evaluated: following instructions, spatial reasoning, general manipulation and robustness.
Understanding is not executing
All the models have the same profile: excellent in following instructions (three are close to a perfect score), solid in spatial reasoning, but lagging behind in handling. GPT-6 Astra stands out in terms of instruction following and robustness, while Claude Opus 5.5 takes the lead in spatial reasoning, manipulation and overall ranking. The authors see complementary strengths and a persistent gap between the understanding of a task and its reliable physical execution.
The study also shows that harness design affects success rate, planning cost, and execution time. Case studies conducted with Astra explore whole-body coordination, switching from one robot to another, precision handling, dexterous hands, and tasks on real robots.
Read the full article on the original source
Go to the source →