Prompt injection: the model only ‘decides’ to disobey in the final third of its layers, and this can be prevented in 77 to 92 per cent of cases

By analysing five models, each with between 4 and 32 billion parameters, layer by layer, researchers have identified a late-stage bottleneck where compliance with the attacker’s demands is determined.

Many works measure the frequency of success of quick injections; This one looks for the location of the network where the model switches. The authors apply causal "patching" of activations, layer by layer, on five models ranging from 4 to 32 billion parameters.

Visible early, decisive late

As a result, the attack information is readable from the first layer, but it has almost no causal effect on behaviour up to a neck located in the last third of the network. Changing activations at this location cancels obedience to malicious instruction in 77-92% of cases. The mechanism stands in a compact linear subspace — ranked 8 for models of 4 and 14 billion, 64 for those of 32 billion — and remains stable from one model family to another.

The same point is also the best place to detect an attack: a detector placed there resists better than those of the first layers to surface makeup, such as replacing letters with numbers (lettpeak).