A test suite comprising 18 tasks and 3,100 demonstrations on a Unitree G1 reveals a gap between selecting a tool and the complete execution of the task.
To exceed their physical limits, humanoid robots will have to handle tools, which means choosing the right one, then coordinating manipulation and displacement. No test bench evaluated these capabilities together on a humanoid. Researchers propose HumanoidToolBench: 18 tasks spread over three scenarios, three levels of execution and two methods of tool selection, accompanied by ToolBook, a game of 3,100 demos collected in simulation and on a real G1 Unite.
Seven control policies were tested in simulation and three on the physical robot. The observation is clear: knowing the right tool does not guarantee that the task will be carried out, and the gap between the two remains significant. Targeted tests on Nvidia's GR00T N1.7 also show a declining selection accuracy against tools never seen, as well as a tendency to continue execution even when the set has nothing to do with it.
The code and data are published in open access so that other teams can measure their progress on the same basis.
Read the full article on the original source
Go to the source →