Indexing robots: OpenAI, Google, Microsoft and Anthropic adopt a common signal to refuse AI training

Offered to the IETF by Anthropic, a "policy token" placed in the robots.txt will allow publishers to refuse the training of several AIs with one gesture, instead of blocking each robot separately.

Until now, site publishers had to express their preferences robot by robot, company by company. OpenAI, Google, Microsoft and Anthropic announce that they will respect a shared signal, recorded in the robots.txt file, which will apply to everyone at once: if they encounter it, their robots will refrain from vacuuming the site.

This "policy token" was proposed to the Internet Engineering Task Force by Kevin Kelley, technical manager of standards at Anthropic. Anthropic is already applying it; Google and Microsoft will integrate it next year, OpenAI "in the coming months". Microsoft also promises to distinguish between the refusal of training and that of grounding, with no effect on ranking in Bing. Meta supports the initiative without committing to honor it.

The announcement is in addition to the tools already existing on the publisher side, such as the RSL standard which allows robots to be paid, or the blocking by default of AI robots at Cloudflare.