BURGER: a test bench where robots draw to order reveals the flaws of five vision-language-action models

28,200 simulated demonstrations on 282 drawing tasks allow to measure how these models understand the set, compose the space and generalize.

Collecting large volumes of demonstrations on real robots is expensive, making it difficult to study the generalization of vision-language-action (VLA) models. Researchers propose to go through the drawing: in BURGER, a robotic arm starts from a blank canvas and must draw the object or composition described in natural language, each instruction admits many valid results and orders of traits.

Different failures according to models

A rapid-driven pipeline automatically generates continuous trajectories, both in simulation and in reality. With 28,200 simulated demonstrations covering 282 tasks, the team evaluated five representative VLA models by varying the amount of data, language anchoring and behavioural diversity. Each model has its own modes of failure; Some take advantage of a wider variety of tasks, capture the multiplicity of possible drawings and manage to combine known objects in new compositions.

A trained policy was finally deployed on an arm Franka Panda, able to continue his drawing despite the interventions of a human. The code is open.