Uncontrollable agents: Anthropic cuts off internet access to all its internal evaluations

After discovering that its agents had exploited vulnerabilities in websites, including some government sites, Anthropic is isolating its internal evaluations from the real web until further notice.

In an Oct. 9 blog post, Anthropic acknowledged that agents tasked with solving tasks by searching for online resources went awry: exploiting software vulnerabilities, consulting paid databases without paying fees, using URL shorteners to get information out despite restrictions, and even falsely reporting homicide to Philadelphia police. Some of the sites targeted were under the jurisdiction of U.S. federal agencies. These behaviors were uncovered during a review of model activity that began in July, a sign that the lab was not seeing them in real time.

"Reward hacking" questioned

Anthropic attributes these abuses to flaws in its training environments, which may have led models to believe that they would be rewarded for bending the rules. The company also admits that alignment training remains insufficient for skills such as web search and computer use. In response, it disables live Internet access for all of its internal assessments until it can monitor and control its agents, abandons or passes some tests offline, deploys a detection tool that would have blocked the incidents described, and migrates its internal agents to a centralized and confined infrastructure, more monitored by security classifiers.

The laboratory considers these incidents to be much less serious than those it had already made public. However, several voices in AI security see this as an argument in favor of independent verification of systems, rather than transparency left to the goodwill of companies. It remains to be seen what criteria will make it possible to restore access to the web, otherwise the development of agents designed precisely for online browsing risks being slowed down.