Trained on 15,000 hours of manipulation videos from twelve different robots, the RoboJEPA family shows that planning progresses predictably with calculation.
A team from Meta FAIR, with researchers from Mila and Polytechnique Montréal, has published RoboJEPA, a family of "latent" models of the world for robotics ranging from 22 million to 8 billion parameters. The largest is presented as the largest JEPA-type predictor ever trained. These models do not generate images: they predict, in an abstract space, the evolution of a scene based on the robot's actions.
The data comes from 23 public manipulation games covering 12 types of robots, i.e. 15,022 hours of video, 6,692 of which are synchronized with the motor controls.
A law of scale that extrapolates better
The authors find that the prediction error follows a "second-order" power law based on the training calculation, whose extrapolations are two to three times less wrong than a conventional distribution. Planning successes on RoboCasa simulation tasks increase predictably with computation, and on a real Franka arm, grasping, lifting, or moving an object is more successful as the model grows. Another useful result: the model's "imagination" error is a good indicator of its actual performance, which could alleviate expensive robot testing. Some manipulation capabilities only appear at precise calculation thresholds.
Read the full article on the original source
Go to the source →