LWN details AI scrapers overwhelming infrastructure through residential proxies

Category: Engineering post
Status: discussion
Sources: LWN (Jonathan Corbet), HN discussion
Summary: Jonathan Corbet wrote on 2026-07-10 that AI training-data scrapers increasingly route requests through residential-proxy networks, software installed on ordinary users' devices that fetches pages on command, spreading load across millions of IPs and defeating IP-based blocking. He reports git forges and mailing-list archives among the hardest hit, and that LWN saw its heaviest scraper attack on 2026-07-02, correlated with the takedown of a residential-proxy network. Cited mitigations include proof-of-work challenges such as Anubis, rate limiting, and caching.
Why it matters: Residential-proxy scraping turns automated crawling into a distributed load problem that per-IP defenses cannot address, raising the operational cost of running public git and archive infrastructure.