The same evaluation setup can cause a model’s score to vary by more than 80 points

Audit of eight out of ten cybersecurity test suites: the published rankings depend largely on rarely documented choices of test cases.

Four researchers — Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah and Yazan Boshmaf — have audited eight sets of cybersecurity tests on ten language models, treating evaluation not as a measure but as a treatment chain. Their conclusion is in the title: published scores depend on the configuration of this chain as well as the model evaluated.

The audit identifies 15 systematic failure modes. A single pipe choice can shift the score of a model of more than 80 percentage points and reverse comparisons between models. When the authors apply a uniform evaluation procedure to all test sets, nine of the ten models change by at least three places in some tests.

The authors draw from this a methodological recommendation: to consider the audit of the evaluation chain as a basic requirement, and not as an optional refinement, when we purport to compare models on safety tasks. The work was tabled on September 8, 2026 on arXiv.