• Sources: primary
  • Summary: Cloudflare introduced a system that continuously evaluates whether Verified bots follow robots.txt and reduce request rates after HTTP 429 responses, producing an explainable trust band. Cloudflare is using it internally now and plans to publish scores on Radar. It also plans to let customers choose the trust threshold for handling Verified bot traffic.
  • Why it matters: Crawler operators behind Cloudflare sites now need to honor 429 backoff and robots.txt consistently to keep verified status.