Models guess "who speaks" from the content and structure of the text. Injection training only protects against half the problem; Internal probes detect the attack.
A prompt injection can hijack an agent who is reading a web page or the result of a tool. Ezra Edelman, Benjamin Edelman, Surbhi Goel, and Eran Malach are interested in a specific variant, which they call role spoofing : malicious text disguises itself as a message from the user, exploiting the tendency of models to assign a passage to a caller based on their appearance rather than the role officially associated with them.
Content and form
The researchers show that the models rely on two types of clues to know who said what: content (meaning, writing style) and structure (control tokens, formatting). Attacks that combine false structure and fake content are particularly effective. In a simplified experimental setting, adversarial training that teaches the model to ignore content-based injections neutralizes content-based injections well, but allows structure-based attacks to pass, which could explain why state-of-the-art models, better armed against many injections, remain vulnerable to role spoofing.
As a countermeasure, the team proposes linear probes trained to predict, from the model's internal activations, the role of each segment of the conversation; they effectively spot spoofing attempts in the different scenarios tested.
Read the full article on the original source
Go to the source →