Scaling with safety: Cloudflare's approach to global service health metrics and software releases

Cloudflare describes Health Mediated Deployments (HMD), a data-driven system that automates safe rollouts by monitoring service health via Prometheus and Thanos, using backtesting, recording rules, and distributed query execution to detect regressions and revert changes early. The post details architectural optimizations—pre-aggregation, distributed queries, adaptive priority-based concurrency control, and R2 location hints—that reduce global query latency and cut batch runtimes by 15x, while prioritizing on-call workloads. It also introduces an experimental Parquet-based time series storage proof of concept aimed at improving object storage read patterns and large-scale observability.