Since September 15, a robot that does indexing, collecting for agents and building corpora at the same time has been refused by default on pages carrying advertising.
The deadline set by Cloudflare has expired: since September 15, 2026, the host's default settings refuse so-called mixed-use robots on any page with advertising. This applies to user agents that confuse three distinct functions — indexing for traditional search, retrieving pages on behalf of conversational agents, and collecting content for model training.
The reasoning put forward by the company is based on a distinction that publishers have been demanding for years: to remain visible in search engines, without gratuitously feeding training corpora or response engines. A correctly declared robot, whose purpose is explicit and separate, continues to pass.
The change applies to new customers, sites added by existing customers, and all free accounts; configurations already in place with paid customers are retained, and any site owner can manually adjust the setting. OpenAI, Anthropic, Google, and xAI were already leveraging separate user agents for training and on-the-fly recovery, limiting the immediate effect for major labs.
Read the full article on the original source
Go to the source →