EgoTools: 100 hours of first-person video to teach embodied AIs how to use tools

The new dataset and its test set of 1,000 questions reveal that the best video models still struggle to contextualise the use of a tool within what they see.

Screwing, cutting, measuring: a large part of physical tasks, both domestic and professional, involve a tool. For a robot or an embodied agent to understand them, it must understand how the hand, the tool and the object are articulated, where the procedure is and what effect each gesture produces. Multimodal video models, which are good for describing a scene, remain weak in this field due to a lack of suitable real data.

A corpus and a test bench

EgoTools, presented on arXiv on September 30, 2026, fills this gap with two bricks. EgoTools-Data brings together 100 hours of recordings filmed from the user's point of view, with synchronized sound, detailed captions, reasoning-centered narrations and 3D information. EgoTools-Bench offers 1,000 questions and answers divided into four parts, from perception and geometry to procedure and causality.

The best model tested, Gemini-3.1-Pro, achieved 66.9% overall success but only 51.7% on perception and visual anchoring. Used as training data, the corpus increases Qwen3-VL-8B-Instruct from 50.0% to 60.9% on the test bench, with no overlap between training and test videos.