REDAgentBench: Nearly one in five successful attacks occur after the agent has stated the rule they are violating

Out of 1,661 attack cases and 65.69% average success, the authors uncovered a gap between recognizing a security constraint and sticking to it.

Measuring the security of an LLM agent by the success rate of an attack alone doesn't say much about what actually happened. REDAgentBench, filed on August 11, 2026, offers a more demanding method: deriving attacks from explicitly formulated security constraints, running them in isolated sandboxes, and then verifying harmful effects from service receipts and end-state changes—in other words, seeing the damage rather than assuming it.

The protocol covers 1,661 attack cases across five service surfaces, evaluated on six models and three agent harnesses. The average macro success rate is 65.69%.

The gap between recognition and execution

The most striking result is what the authors call the "Recognition--Execution Gap": nearly one in five confirmed attacks occurs after the agent has explicitly stated the relevant constraint. The model knows, says it, and transgresses anyway. The good news, however, is that a simple reminder of the security policy, without any retraining, causes confirmed violations to drop by more than 70 percentage points.