Our billing pipeline was suddenly slow. The culprit was a hidden bottleneck in ClickHouse

Cloudflare engineers describe how changing a ClickHouse partition key to `(namespace, day)` to enable per-tenant retention massively increased the number of data parts, which unexpectedly caused severe lock contention in query planning and slowed time-critical billing jobs. Through tracing and flame-graph analysis they implemented three optimizations — switching to a shared lock, deferring copies with a shared cached parts list, and adding a binary-search prune by `namespace` — upstreamed the fixes, and restored stable query performance despite continued growth in part counts.