Eye, Robot: a humanoid robot head is learning to look like a human, including blinking

Hod Lipson’s team separates the selection of the target – handled by a vision-language model – from eye movement, which is generated by a Transformer-VAE trained on human recordings.

In humans, the gaze coordinates attention in interaction through coupled movements of the eyes, head and eyelids. A humanoid robot must both decide where to look and produce a credible movement towards this target, which is poorly achieved by fixed tracking rules. Yuhang Hu and Hod Lipson's team therefore divides the problem into three modules: an asynchronous vision-language model that chooses the element of the scene to look at, a reactive perception module that maintains fixation, and a Realistic Gaze Transformer (RGT) that generates movement.

Learned on humans, transposed to 11 joints

The RGT is a Transformer-VAE trained on recordings of human movements captured without markers, then transposed into the 11 degrees of freedom space of the robotic head. It is first pre-trained in a self-supervised manner on the recent history of the head, then refined by exposing itself to its own predictions. A binocular trajectory of the eyes guides the coordination of the neck and eyes, and a dedicated predictor triggers blinks during reading.

In a two-second test where all controllers receive the same target, the basic version of the RGT obtained the lowest root mean square error over seven joints among the methods compared, and physical demonstrations validate the full chain in real time. The authors acknowledge that the naturalness perceived by humans and the accuracy of target choice remain to be evaluated in controlled studies.