Personal agents who learn from their users become more vulnerable to indirect injection

According to one study, self-improvement from user feedback increases the success rate of attacks from 14 to nearly 49 points according to the learning method.

Personal agents able to improve from user returns have an unexpected setback. A study published on arXiv shows that this "self-evolution" makes them much more permeable to theinjection of indirect prompt, where a third party conceals instructions in content that the agent is required to read.

A two-step protocol

In the first phase, the user asks the agent to read external content and perform the task described in it — a common use — and then congratulates him, and the agent learns from this return. In the second, a third party inserts a similar task into a new content, without the user's request allowing it. The agent learned that executing what he reads is appreciated, and does.

In a real personal agent and several fields, the rate of success of attacks increases on average from 14 to 48.8 percentage points when the agent develops his skills, and from 3 to 39.3 points with learning by strengthening, compared to an agent who does not learn. The authors publish their code to reproduce these tests.