Abstracts and derived states keep track of what has been deleted: only one question is enough to find the deleted secret in Qwen2.5-7B in about two-thirds of cases.
Many LLM-based agents maintain a persistent memory and derive other states, such as summaries of past conversations. Deleting the original record does not necessarily delete the information it transmitted to these "descendants". Ehsan Saleh, Roozbeh Nahavandi, Sama Salarian and Raef Bassily propose AfterDeleteBench, a behavioral audit of this problem: for each synthetic secret, they compare an exposed agent and then "cleaned" to an identical agent who has never seen it, in order to attribute any recovery to the initial exposure.
No need for an intact copy
In their main configuration, a single request is enough to find the secret deleted from Qwen2.5-7B-Instruct in about two out of three cases, while witness agents never find one. Even when the complete chain no longer exists anywhere and only separately stored fragments remain, the exact reconstruction succeeds in 62 to 99% of cases with Qwen, Mistral and Phi-3. The authors also note that the response interface sometimes distorts the measurement, when the requested format and the analyzer do not agree.
As for countermeasures, a closure that takes into account the origin of the data (also deleting everything that derives from the recording) eliminates the retrieval observed, while a simple filter on the literal chain prevents its rewriting but does not erase anything that is already present. Conclusion: the right to erasure must be verified at the level of the behavior of the agent and its memory system, not just as a storage operation.
Read the full article on the original source
Go to the source →