Cloudflare adds a Disallow AI Training setting and changes Block to cover mixed-use crawlers including Googlebot, Bingbot and Applebot

Sources: primary, HN submission
Summary: In a post dated 2026-09-15, Cloudflare states that from that date the Block and Block on pages with ads settings apply to mixed-use crawlers, which it names as Applebot, Bingbot and Googlebot, so either setting now affects search indexing as well as training. The new Disallow AI Training setting publishes the no-training preference in robots.txt through Bot Preference Sync, keeps accountable mixed-use crawlers allowed for search, and blocks every other training crawler, including the training-only crawlers Cloudflare names from Amazon, Anthropic, Meta and OpenAI. Block AI Bots is deprecated in favour of the Search, Training and Agent controls, Managed Robots.txt is deprecated in favour of Bot Preference Sync, and Cloudflare states existing preferences migrate automatically to preserve their practical effect.
Why it matters: The setting that used to stop AI training without touching search no longer behaves that way, so a fresh bot configuration can drop a site out of search indexes.
Follow-up: Track whether Apple, Google and Microsoft meet the time-bound commitments Cloudflare counts toward its Accountable requirements.