Introducing pay per crawl: Enabling content owners to charge AI crawlers for access

Cloudflare introduces Pay per crawl (private beta), a framework that lets publishers control and monetize AI crawler access via `HTTP 402 Payment Required`, authenticated crawler identities (Web Bot Auth with Ed25519 and HTTP Message Signatures), and pricing headers (`crawler-max-price`, `crawler-exact-price`, `crawler-charged`). Domain owners can set allow/charge/block policies enforced after existing WAF/bot controls, while Cloudflare acts as merchant of record to aggregate billing and payouts. The initiative aims to give creators programmatic control over access and pricing today and to enable future agentic, dynamic, and granular licensing models for content use.