Cloudflare incident on June 20, 2024

On June 20, 2024, Cloudflare experienced a 114-minute service degradation affecting multiple data centers, peaking at 2.1% HTTP errors and a 3x increase in p99 TTFB. The impact stemmed from two independent events: automated network monitoring prompted suboptimal traffic re-routing that congested backbone links, and a newly deployed DDoS mitigation exposed a latent bug in rate limiting that caused certain HTTP requests with mixed cookies to trigger an infinite loop, poisoning request-handling processes and driving CPU saturation. Recovery involved automated and manual service restarts, Traffic Manager fixes, global rollback of the DDoS rule, and subsequent capacity and design/process improvements; Cloudflare found no evidence of a breach or intentional exploitation.