IronMind: 10,000 hours of eye-level video footage has increased a humanoid’s success rate on novel tasks from 11.7 per cent to 55 per cent

This vision-language-action model learns fine motor skills from first-person human videos, by representing the gestures in the camera’s coordinate system rather than mapping them onto the robot’s body.

Videos shot by humans equipped with a front-facing camera are a wealth of data for teaching robots how to manipulate objects, but there are two obstacles to their use: human hands don't look like robotic grippers, and these cheap recordings don't capture the torso posture needed for traditional motion transfer methods. In addition, there is often noisy hand tracking and rough textual annotations.

Act in the camera cue

IronMind, presented on arXiv, gets around the difficulty by dispensing with any reconstruction of the body: actions are described directly in the space of the camera, that of the video itself, and the dimensions of action of the human and the robot are semantically aligned. The model is pre-trained on a mix of human videos and heterogeneous robotic data, with budgets ranging from 250 to 10,000 hours; the loss of validation decreases in a roughly log-linear way with volume.

The gain can be seen especially on a real robot, after post-training, facing six tasks with objects, uses and instructions never seen before: the 10,000-hour model succeeds in 55% of the tests, compared to 11.7% at best for all budgets up to 5,000 hours and 5% without pre-training. At the same volume, the representation in the camera cue also beats that propped up on the chest.