Caching KMS data keys in multi-thread environments: Per-tenant encryption for event-driven systems at scale

This post explains how envelope encryption at multi-tenant, high-throughput scale can cause a cache stampede that drives millions of redundant AWS KMS GenerateDataKey and Decrypt calls, increasing cost and latency, and presents two mitigation approaches: (A) AWS-recommended hierarchical keyring with DynamoDB branch keys to coordinate cache refreshes across instances, and (B) a custom CachedKmsClient using Caffeine LoadingCache to enforce single-flight KMS API calls per region. NICE Actimize implemented the custom caching approach and reported a 77% reduction in KMS costs while maintaining per-tenant cryptographic isolation and simplifying operations.