Control content use for AI training with Cloudflare’s managed robots.txt and blocking for monetized content

Cloudflare announced two tools to help website owners control AI crawlers: a managed robots.txt that automatically signals AI training bots (e.g., Google-Extended, Applebot-Extended) not to use site content while preserving SEO, and an option to block AI bots only on pages that serve ads using bot identification and ad-detection heuristics; backed by Radar data showing extreme crawl-to-referral ratios for AI bots (OpenAI ~1,700:1, Anthropic ~73,000:1) and shifting bot traffic patterns, these features are available to all customers and will be kept current as the AI bot ecosystem evolves.