Legislation
Cloudflare blocks by default crawlers that mix search and training
Since September 15, a robot that does indexing, collecting for agents and building corpora at the same time has been refused by default on pages carrying advertising.