Why we're rethinking cache for the AI era

Cloudflare analyzes how AI crawler traffic—characterized by high unique-URL ratios, content diversity, and inefficient crawling—significantly increases cache misses, origin requests, and bandwidth costs, degrading human user performance. The post models agent behavior, cites real-world impacts on sites like Wikipedia and Fedora, and proposes mitigations such as AI-aware caching algorithms, separate cache layers for AI traffic, traffic filtering, rate-limiting, and product controls like AI Crawl Control and Pay Per Crawl.