- Sources: Cloudflare blog, HN 49052564
- Summary: A Cloudflare post dated 2026-07-01 and last modified 2026-07-15 resurfaced on Hacker News on 2026-07-26. It replaces the single "block AI bots" preset with per-behavior controls for Search, Agent, and Training crawlers, available down to the Free tier, inside a wider bot taxonomy that also names Transact, Data Collection, Security Testing, SEO, Ads Verification, Social, Feed Fetching, and Monitoring. From 2026-09-15, new domains onboarding to Cloudflare get Training and Agent blocked by default on pages that display ads, with Search allowed. From the same date, multi-purpose crawlers are evaluated against all their behaviors under the most restrictive applicable rule, so Googlebot, Applebot, and Bingbot are blocked for customers who block Training, unless the customer opts out in Security settings beforehand. A
use field with values immediate, reference, and full extends Content Signals in managed robots.txt as a stated preference rather than an enforced block, Enterprise Bot Management gains a searchable bot directory called BotBase, and Cloudflare proposes carrying operator identity through intermediaries in the RFC 7239 Forwarded header, for example Forwarded: for="openai";use="reference". - Why it matters: Anyone shipping an agent that fetches web pages should assume a growing share of Cloudflare-fronted sites will classify that traffic separately from search crawling and block it by default.
- Follow-up: Watch whether the 2026-09-15 defaults ship on schedule, whether the
use Content Signals field gains adoption outside Cloudflare, and whether other CDNs copy the per-behavior taxonomy.
send feedback on this story