Quadruped robots: a refined LLM decides which obstacle to push and where, and succeeds in 34 out of 45 sails on the real robot

Researchers propose a natural language planner that chooses skill, object and target, coupled with a push policy that achieves 93.3% success in simulation.

A robot that navigates generally assumes that a free path exists. In a room cluttered with cardboard boxes or trolleys, it is sometimes necessary to push an obstacle to get through. A team of five researchers entrusts this decision to a language model refined by LoRA on synthetic data: from a natural language instruction and a structured description of the scene, it produces a sequence of parameterized actions – which skill, on which object, towards which position – which are then executed by reinforcement learning policies.

Better than a state-of-the-art model solicited by prompt

This planner goes beyond a classic geometric planner and a state-of-the-art model simply queried by prompt on interactive navigation tasks. Comparisons show that with a well-structured representation, linking skills does not require pre-training, but that it makes geometric targets more reliable. On the low-level side, a planar push policy mobilizing the robot's entire body succeeds in 93.3% of simulation tests, compared to 83.5% for a hierarchical approach, and finishes more than 60% faster.

On the real robot, the hierarchy chooses the right combination of skills in 44 out of 45 trials and completes the task in 34 cases.