RoboWorld: Code agents manufacture 25 physical work simulations, and best models fail on all test tasks

Gym-Anything for Robotics entrusts the creation of simulators to code agents, guided by US GDP. Average peak model score: 0.06.

Realistic robotic simulations would allow for massive training of embodied models, but building them requires a lot of human expertise and doesn't scale up. The Gym-Anything for Robotics framework is based on a simple observation: writing a simulator is a programming task, so every progress of code agents should result in richer simulations.

The authors choose the jobs to be simulated based on US GDP data, to cover physical work with high economic value, and subject each simulation to a contradictory loop of creation and audit. The result: RoboWorld, 25 simulations with 25 test environments and 175 reinforcement training environments. No state-of-the-art model passes any test task, with an average reward of 0.06.

The team proposes several areas for improvement, all of which are effective: in-context learning on repeated attempts, a "teacher" who reads the internal state of the simulator, or policies written in code to generate demonstrations intended to train vision-language-action models such as π0.5. Pipeline, environments and benchmarks are published.