Shrinking the haystack: Building a cloud threat detection engine

This article provides a practical blueprint for building a scalable cloud threat detection pipeline, detailing choices across runtime frameworks (e.g., Kubernetes), programming languages (Ruby to Go migration), messaging architectures (S3 → SNS → SQS), and infrastructure-as-code (Terraform) to enable modular, cost-effective, and resilient processing. It emphasizes independent, composable components, easy data replay, extensibility for experiments (e.g., data lakes, generative AI), and operational guardrails such as monitoring, alerting, and autoscaling to handle rapidly scaling cloud telemetry workloads.