By alternating the lighting, the sensor alternates to capture an image through its membrane and a touch print, from the same point of view and without calibration between cameras.
The external cameras of a robot are often masked precisely where the hand touches the object. The team of Xiangdong Wang, Nima Fazeli and Xiaonan Huang offers with GelSlim 5.0 a finger that does both at the same time: a fixed camera and a continuous UV-sensitive contact layer. Depending on the lighting on, the camera sees either the object through the membrane or the tactile distortion of the membrane. Since the two images share exactly the same angle, there is no longer any need to align two sensors.
Merge without overwriting touch
To combine these signals, the authors designed VisTacFusion, a multimodal transformer with two prediction paths: one for the position of the object on the plane, the other for the geometry of the contact (depth and surface normals). The fusion is deliberately asymmetrical so that the visual context enriches the tactile map without erasing the spatial detail, and a random abandonment of the modality during training makes it possible to work with only one of the two images.
On 12 real objects, combined use reduces depth error by 61%, angular error by 72% and translation error by 61% compared to touch alone. Compared to vision alone, the gain is especially spectacular on depth (−94%), more modest on angle and position.
Read the full article on the original source
Go to the source →