Benchmarking the Agentic SOC: How we evaluate LLMs for security workflows

This document describes an evidence-based evaluation framework for LLM-driven SOC agents in Elastic Security: it seeds a reproducible synthetic intrusion (the Chrysalis DLL side‑loading scenario using an EICAR test hash) into a live deployment, runs 21 prompts across seven capability categories plus two additional suites (Attack Discovery and Automatic Migration), captures full execution traces and tool calls, anonymizes outputs for blind scoring, and grades models on both answer quality and correct tool/skill selection to ensure agents are judged on grounded actions rather than fluent but unsubstantiated prose.