Jailbreak: Obfuscation deceives the seven open models tested, and a simple reminder in the prompt divides successful attacks by nine

On Llama, Mistral, Qwen, Gemma and Phi, the obfuscation averaged 54.3% of attacks; the model size does not predict its robustness.

An evaluation published on arXiv compared seven language models — Llama-3-8B, Llama-3.1-8B, Llama-3.2-3B, Mistral-7B, Qwen2.5-7B, Gemma2-9B and Phi-4-14B — to five families of jailbreak: direct demand, role play, hypothetical framing, crushing instructions and obfuscation.

Obfuscation, the most transferable attack

Hide malicious query is the most effective method: 54.3% average success rate, and no model escapes it completely. The authors do not observe a significant relationship between model size and robustness, and note that models in the same family have very different vulnerability profiles.

Four defenses were then tested on the two most fragile models: blacklist of keywords, prompt security reminder, query reformulation and perplexity filtering. The safety recall is the most regular: it increases the success rate of attacks from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B, without any false positive or additional inference costs. The code and data sets are published.