Cloudflare Overhauls AI Bot Management
Cloudflare has announced a significant shift in how website owners can manage AI bot traffic. Since July 2024, Cloudflare customers have been able to block all AI crawlers with a single click. Now, instead of a blanket block, site owners can choose between three categories: Search, Training, and Agent. This change provides more nuanced control over how AI bots interact with websites.
New Granular Controls
The new AI bot controls in the Cloudflare dashboard allow site owners to manage access for Search, Training, and Agent bots separately. Search covers search engine indexing, Training covers data collection for AI model training, and Agent covers bots acting on behalf of users, like ChatGPT. Starting September 15, 2026, Training and Agent bots will be blocked by default on pages that carry ads, since advertising signals that a site wants human visitors. Search crawlers will still be allowed. Multi-purpose crawlers like Googlebot that both search and train will be treated according to whichever rule is strictest. The new options are available to users on the free plan as well.
Bot Operators Encouraged to Split Crawlers
Cloudflare is pushing bot operators to split their crawlers by purpose. Cloudflare CEO Matthew Prince has previously called out Google for bundling its search crawler with its AI crawler. In June 2026, Prince noted that bot traffic had surpassed human traffic on the internet for the first time, a milestone he originally didn't expect until late 2027.
BotBase: Enterprise Bot Database
For enterprise customers, Cloudflare is also launching BotBase, a searchable database of all known bots built right into the dashboard. It shows how each bot is classified and how it uses content, whether it just links to pages or reproduces them in full. Cloudflare is also changing the rules for verified bots. Previously, all verified bots were allowed through automatically. Going forward, a bot's category determines whether it gets access. The 'Verified' label alone won't be enough anymore: Bot operators will need to prove they identify themselves honestly and don't abuse the access they're granted.
Implications for Website Owners
This update gives website owners more flexibility in managing AI bot traffic. By separating bots into categories, site owners can allow search indexing while blocking training and agent bots on ad-supported pages. This helps protect content from being used for AI training without permission while still allowing search engines to index the site. The default block on Training and Agent bots for ad-supported pages is a significant change that aligns with the interests of publishers who rely on advertising revenue.
Impact on AI Companies
AI companies that use crawlers for training or agent tasks will need to adapt to these new controls. They may need to split their crawlers into separate bots for different purposes to comply with site owners' preferences. Cloudflare's move also puts pressure on other internet infrastructure providers to offer similar granular controls.
Conclusion
Cloudflare's new AI bot controls represent a major step forward in giving website owners more control over how their content is used by AI. By providing granular options and a default block on training and agent bots for ad-supported pages, Cloudflare is helping to balance the needs of content creators and AI developers. The introduction of BotBase for enterprise customers further enhances transparency and control.
This article is based on reporting by The Decoder. Read the original article.
Originally published on the-decoder.com
