KODA: teaching a humanoid robot using first-person videos of humans, without robotic demonstrations for each task

The method converts first-person RGB video footage into vision-language-action training data, achieving a 45-point improvement in performance on tasks not covered by robotic demonstrations.

Training a humanoid in whole body vision-language-action policies usually requires costly robotic demonstrations, with observations seen by the robot and state-of-action labels compatible with its controller. The first-person videos of humans offer much more supervision, but their views and movement indices do not match those of the robot. KODA (Kinematics-Observation Data Alignment), presented by Zibo Zhou and his colleagues, converts these egocentric videos into drive data thanks to a reusable canonical trajectory that guides both the synthesis of robot views and the generation of controller-specific state-action labels.

KODA first generates a simulator-checked whole body trajectory, which directly determines the positions of the on-board camera used to synthesize observations. Movement, observations and instructions form a canonical recording; for each deployment controller, the system re-optimizes the controls to track and label this movement. Evaluated on four tasks of locomotion-manipulation of a Unite G1 with two controllers (SONIC and HEFT), the method improves the average success rate of 45.0 and 46.25 points on conditions covered by human videos but absent from robotic data, with constant robotic data volume.