PRISM manufactures hundreds of counterfactual videos from a handful of real ones, and the humanoid carries objects he has never seen before

The bottleneck of visual imitation is the collection of clean videos. PRISM generates variants, reconstructs them into physically plausible trajectories, and deploys the policy without real robot adjustment.

Teaching a humanoid to move while manipulating — carrying a crate, moving a bin — by visual imitation requires a lot of usable videos: whole body visible, interaction with the object unmasked. This requirement is precisely what blocks scaling. The team of Zihan Wang, Zhen Wu, Pieter Abbeel and Rocky Duan proposes to get around the problem by amplifying a small number of real examples instead of collecting more.

PRISM is a real-to-simulation-to-real sequence. From a few exemplary videos, a video-to-video generation produces hundreds of "counterfactual" sequences of human-object interaction. A reconstruction anchored on the contacts then retrieves the movements of the person and the object, and retargets these imperfect data into physically coherent trajectories. The internal variability of each category of objects is then sufficient to result in a single policy that generalizes to unseen instances of the category.

The policy is deployed on a real robot without any additional adjustments in the physical world, relying solely on on-board depth observations. The humanoid picks up, transports and deposits crates, barrels, bins and balloons, including on never-before-seen specimens, sizes and starting configurations.