By replacing the reference text with an unrelated one, researchers have shown that the verdict of seven models rarely changes, whilst a hidden instruction within the facts can easily manipulate them.
When a language model justifies a legal decision by citing the applicable law or jurisprudence, it is readily assumed that its conclusion follows from it. One team tested this hypothesis by keeping the facts the same but substituting an unrelated reference for the authority invoked, and then reading in the internal states of the model the evolution of its verdict.
To quote is not to rely
Out of seven open models with 8 to 70 billion parameters and four sets of tests covering court decisions and contracts, the models name the correct source in 66.7% to 100% of cases when asked. But their verdict does not change when this source changes in only 0% to 21.7% of cases on CaseHOLD, 30% to 76.7% on ECHR and US Supreme Court cases, and 43.3% to 50% on ContractNLI. Neither size nor a model specialized in legal reasoning bridges the gap.
Conversely, a malicious instruction slipped into the statement of facts is followed in 73.3% to 96.4% of cases. For the authors, the explanations generated should therefore not be used as compliance or audit documents, since they do not prove that the decision is based on the cited text.
Read the full article on the original source
Go to the source →