TamperBench: all 21 open-weight models tested have their guardrails dropped

A team led by the University of Waterloo and FAR. AI subjected the main open models to systematic fine-tuning attacks. None held.

Researchers from the University of Waterloo and the non-profit organisation FAR. AI, supported by MIT, ETH Zurich and the University of Toronto, published on 25 August the results of TamperBench, an evaluation framework that combines attacks on the weights of the model and those on its internal representations in a single protocol. Twenty-one of the most widely used open-weight models were examined. All of them were able to be altered despite the protections integrated by their designers.

The protocol relies on systematic scans of hyperparameters rather than a single attack, and jointly measures the safety and utility of the model after manipulation. The so-called "jailbreak" fine-tuning stands out as the most destructive vector, and defenses applied at the alignment stage do not withstand these scans.

"When the guardrails are removed, the model can be used on a large scale" to cause harm, says Sirisha Rambhatla, while co-author Saad Hossain believes that current defenses "don't seem robust enough." The researchers cite mass disinformation campaigns, elaborate email scams and the production of operating procedures to manufacture dangerous chemicals as concrete risks. The study was presented at the ACM Knowledge Discovery and Data Mining conference in South Korea.