GPT-6 Astra reads images directly from touch sensors and goes beyond specialized robotic policies for contact manipulation

Without dedicated training, OpenAI's model interprets the provided touch images as simple photos and achieves 93.8% success on simulated tasks where specialized models were much lower.

The touch has so far been the reserved domain of specifically trained robotic policies. A team led by Chen Yang shows that GPT-6 Astra can use it directly: "vision" tactile sensors produce RGB images of the deformation of a membrane in contact with objects, and these images can be given to the model by its usual visual input, without any adaptation.

Nineteen simulated tasks

With only a few examples in context and a closed control loop, Astra was evaluated on 19 tasks of UniVTAC and OpenViTac benches, with 210 scene configurations per setting. By combining vision and touch, it achieves 93.8% and 81.8% average success, 22.9 and 13.1 points higher than the best specialized policies compared. Even without touching it, vision alone (76.3% and 80.6%) precedes these references. Trajectory analysis shows active sampling, cautious grip and recovery behaviour after error, but also difficulties when efforts are tangential.

The authors recognize that these test benches appear close to saturation and that the results remain to be confirmed on real robots. They argue for joint vision-touch pre-training and more demanding assessments.