Scaling Security Insights: how we achieved a 10x increase in global scanning capacity

Cloudflare increased Security Insights scanning throughput by over 10x (from ~10 to >120 scans/sec) by reworking its scanning pipeline: consuming Kafka messages in batches with parallel goroutines, splitting slow and fast processing lanes, optimizing bulk inserts into Postgres, switching the API to active-passive to avoid cross-region latency, and improving the scheduler with per-zone timing, randomized scheduling, and adaptive rate limiting—allowing automatic scanning for all free accounts and more frequent scans for all customers.