Reflex: a humanoid Unitee G1 learns how to catch in the flight of launched cartons, guided by its only camera

This three-step learning framework achieves 85.7% simulation success from an on-board vision, and up to 76% of a human's success rate on the real robot.

Catching an object in motion requires anticipating its trajectory and mobilizing the whole body in less than a second, a still difficult combination for humanoids. The authors of Reflex, presented on arXiv, chose a representative exercise: to catch cardboards thrown, as when unloading a truck, without capturing movement or markers on objects.

Learning one skill after another

The method proceeds in stages. A policy driven by learning by reinforcement first learns to catch with the whole body by knowing the exact state of the cardboard. She then learns to estimate the dynamics of the object from delayed and incomplete observations. Finally, a embedded RGB-D visual encoder learns to find this representation directly from the images, without retouching the controller.

In simulation, Reflex achieved 85.7% of receptions with its own camera, compared to 87.9% with privileged access to the cardboard position; Removing visual history reduces success from 35 to 40 points. On a real G1 unit, the robot captures 65% of the cards thrown at 2.5 m and 40% at 5 m, or 76% and 59% of the success rate of humans placed under the same conditions.