Over 700 million events/second: How we make sense of too much data

Cloudflare outlines how its analytics pipeline scales to hundreds of millions of events per second by using adaptive downsampling, weighted buffering, and Horvitz–Thompson estimators to produce accurate metrics with confidence intervals, even during overload. The team identified a systematic sampling bias in one stage that inflated byte-count estimates and corrected it by shuffling data to ensure random sampling; confidence bands are now used internally and exposed in parts of their public analytics APIs.