KaliBench: No open model exceeds 42% exact commands on Kali Linux security tools

8,504 request-command pairs covering 1,642 tools measure whether a model can translate the intent of an executable command line analyst.

Language models are invited to security centres, where they are asked to turn an analyst's intent into a command for a specific tool. But in the command line, a wrongly placed option or a wrongly associated value is enough to cause execution to fail. Existing assessments, focused on knowledge or full agent duties, did not directly measure this competency.

KaliBench fills this gap with 8,504 pairs combining natural language demand with the expected command, covering 1,642 tools from Kali Linux distribution, 23 types of capabilities and 5 phases of a security operation. Each pair was validated by a model, executed in a sandbox and then reread by humans, and the comparison takes into account the aliases of options to remain reproducible.

A small, upgraded model

Of 24 configurations of open, general or specialized security models, none of them exceed 42% of the exact commands when they are not told which tool to use. However, the authors show that the deterministic signals in the test bench can be used as verifiable rewards: after supervised refining and reinforcement learning, an 8 billion parameter model reaches the level of a 685 billion-billion expert blend model.