How we built high speed threat hunting for email security · Blog · Sublime Security

A technical overview of Sublime’s architecture for high-performance email threat hunting and backtesting: MQL queries are compiled into a candidate-selection SQL phase against partial MDMs in warm storage, then an evaluation phase deserializes only required fields from flatbuffer MDM blobs in cold storage to run expensive enrichments (e.g., ML logo detection). The post explains period chunking, two-tier caching, and pragmatic PostgreSQL workarounds (dead tuples, MVCC update churn, processor coordination), and describes future optimizations such as lossy data structures and dynamic processor scaling to maintain speed at scale.