Google’s AI advantage: why crawler separation is the only path to a fair Internet

Cloudflare’s response to the UK CMA consultation argues that while the CMA correctly recognizes publishers’ lack of control over Google’s use of web content for generative AI, the proposed remedies are insufficient. Cloudflare presents data showing Googlebot crawls far more unique pages than other AI crawlers, warns that Google’s combined search/AI crawler gives it an unfair competitive advantage and undermines publishers’ ad-supported models, and urges the CMA to mandate separation of Googlebot into purpose-specific crawlers so publishers can opt out of AI uses without losing search traffic.