VioLA: a humanoid robot learns to follow instructions from videos of humans, achieving a 100 per cent success rate in locomotion without any fine-tuning

By predicting body and hand movements rather than joint commands, the AI can be trained on 140 million images, 93 per cent of which depict humans.

Making a humanoid robot obey instructions with its whole body comes up against two obstacles: an immense space of action, where legs, arms and fingers must move together without losing their balance, and the rarity of demonstrations carried out on humanoids. Current generalist policies must therefore be retrained by teleoperation for each new task.

Change what the model predicts

The authors of VioLA get around the problem by having the model predict latent representations of body and hand movement, which are then executed by pre-trained controllers. As human and robotic movements are projected in the same latent space, a simple video of a person directly becomes training data: the corpus reaches 140.6 million images, 93.2% of which are of human origin.

On the real robot, VioLA follows movement instructions without any specific adjustments with 100% success, compared to 16.7% for Nvidia's GR00T N1.7 and 0% for Ψ0. In manipulation, it achieves 88.6% success, again without any adjustment per task. The approach works with three different basic architectures; the code and models must be published.