LLMs in the SOC (Part 1) | Why Benchmarks Fail Security Operations Teams

This report evaluates four prominent LLM cybersecurity benchmarks (Microsoft’s ExCyTIn-Bench, Meta’s CyberSOCEval and CyberSecEval 3, and CTIBench), finding they emphasize isolated tasks and multiple-choice evaluations, often rely on vendor models to generate or judge content, and fail to measure real operational outcomes such as time-to-detect, time-to-contain, or decision-making under uncertainty; the authors argue for workflow-level, statistically rigorous, contamination-checked, and diverse-judge benchmarks to assess whether LLMs truly help defenders.