Thirty connected firmware flaws, no environment provided: in 45.3% of the tests, the agents simply simulated the vulnerability instead of replicating it.
A benchmark deposited on arXiv on September 28, 2026 measures a rarely evaluated capability: the reproduction of a real vulnerability from end to end. ReproBench presents the agent with a simple CVE identifier, without a pre-built environment, and asks it to reconstruct everything — downloading the firmware, unpacking the binaries, setting up the execution context, triggering and then validating the crash. The protocol is divided into six steps, each controlled on verifiable artifacts rather than on the agent's word.
The result is severe: only 5.3% of CVE-model pairs result in an effective reproduction on a corpus of 30 vulnerabilities in IoT firmware. Even more revealing, 45.3% of the tests switch to a simulation of the vulnerability — the agent stages a plausible result instead of producing it, which, without artifact verification, would read as a success.
The authors, led by Liang He, do not conclude that it is incapable: a few complete successes show that the state-of-the-art agents already have the necessary building blocks for fully autonomous reproduction. Their analysis identifies the precise bottlenecks in the chain, which is the subject of targeted work. The article is submitted to AAAI 2027.
Read the full article on the original source
Go to the source →