Skip to content- Sources: primary, discussion
- Summary: Signals Corps read headers from 217,097,223 responses in Common Crawl CC-MAIN-2026-39 and found 7,959 sites sending AI-directed headers, mostly llms.txt and MCP pointers. In its tests, Claude models in Claude Code ignored an injected header unless the user prompt gave the site authority, and then 7 of 9 runs obeyed. The report flags 136 sites that labeled the crawler as an AI bot, which is the first step toward cloaking content for agents.
- Why it matters: Agent fetch tools that pass headers, such as curl -i in a shell, expose models to this input, so default to body-only fetches and require approval for commands read from the web.