Across four real-world tasks, vision-language-action policies trained solely on filmed human demonstrations from a first-person perspective perform on a par with those trained using teleoperation, whilst incurring lower data collection costs.
Filming people is far cheaper than remotely controlling a robot to show it what to do. Previous work on transfer from first-person videos had mainly focused on varying the settings, by controlling the legs and arms separately. EgoHumanoid-V2 tackles movements where the whole body must coordinate whilst moving and manipulating objects.
At the heart of the method is a two-step action alignment process: a kinematic correction of the human reference, followed by a refinement that takes the robot’s dynamics into account. Hand position becomes more precise without disrupting overall coordination. To reduce the visual discrepancy between a human arm and a robotic arm, the team superimposes robotic arms onto the images and increases the number of augmentations during training, which also makes the system less sensitive to the viewing angle.
Across four real-world tasks, the policies achieve direct transfer – without any robotic demonstration of the target task – with scores comparable to those achieved through teleoperation.
Read the full article on the original source
Go to the source →