LLM-driven robots: in the face of physical failures, the model does as well as a scripted planner, it is the JSON format that betrays it

The Alpha Fault Injection Bench shows that the 20-point gap between an agent Llama 3 and a deterministic planner comes entirely from corrupt outputs, not reasoning.

Benchmarks of language model-driven robotics mainly measure overall success under ideal conditions, which masks the origin of failures. The Alpha framework, presented on arXiv, simulates calibrated failures at each stage: sensor delay, gripper slippage, out-of-range targets, and planner output corruption, alone or in combination.

To separate the physical weaknesses from those of the model, the author opposes an agent based on Llama 3 (via Ollama) to a deterministic scripted planner, over 150 episodes with identical seeds and geometries. The scripted scheduler succeeds in 64% of the tasks, the LLM agent 44%. But excluding output corruption, both obtain exactly the same score, 55%, without a single divergent episode: slips, kinematic limits and late sensors penalize them in the same way. The whole gap comes from corrupted outputs, which the scripted scheduler overcomes 100% and the LLM agent never. The weak link would therefore be the serialization and validation interface between the prompt and the JSON, rather than a reasoning deficit.