The crawl before the fall… of referrals: understanding AI’s impact on content providers

Cloudflare Radar introduces an AI Insights metric that compares how often AI and search platforms crawl websites versus how often they send referral traffic, revealing significant imbalances (e.g., Anthropic’s Claude ~70,900:1 and Mistral 0.1:1) and shifting patterns over time; complementary Data Explorer views, APIs, and an expanded Verified Bots directory help publishers analyze referrer/crawler activity, while Cloudflare highlights tools and guidance for auditing and enforcing policies to control or block AI crawlers.