Pre-train a robotic model without a single robot: synthetic mathematical tasks are enough to lay the foundations

A Transformer pre-entered only on synthetic data generated by program manages to predict the actions of real robots on tasks he has never seen.

Data from real robots, which vary in both tasks and morphologies, is considered to be the main bottleneck of robotic foundation models. A team of five researchers wanted to find out how far a model can acquire the necessary structure without ever seeing a robot. They designed three data generation chains that produce, procedurally and at low cost, mathematical abstractions of robotic control, and then used them to pretrain a Transformer to predict actions.

Useful as a complement, not a replacement

Trained only on this non-robotic data, the model obtains non-trivial predictions about real physical tasks that it has never encountered, thanks to a learning mechanism in context. Control-inspired data does much better than generic causal data. A reference model, trained with the same computational budget on real robot data, is still better, but only on the datasets that were used to train it.

The authors conclude that this synthetic data can serve as a basis and complement, but not replace, the costly collection of demonstrations on physical robots.