How CrowdStrike Trains GenAI Models at Scale Using Distributed Computing

This document outlines CrowdStrike’s strategy for building cybersecurity-focused LLMs, detailing scalable training infrastructure on Google Cloud’s Vertex Training Cluster with Slurm, modular data pipelines with synthetic data generation, and distributed training using data/tensor/pipeline/context/expert parallelism. It highlights hardware-aware optimization by comparing attention mechanisms (FlashAttention 2 vs. SDPA) on NVIDIA H100 and B200 GPUs, underscoring that meaningful performance gains require tailoring to specific software–hardware configurations to accelerate next-generation security LLM development.