We burned 11.7bn tokens to find the best cyber AI model

This benchmark evaluates 10 AI models (three runs each) against 32 recently disclosed CVEs in source repositories, comparing pooled and single-run recall, precision, exploration behavior, and cost; DeepSeek V4 Pro achieved the highest pooled recall (28/32) while open-weight models closed much of the gap with frontier models, and repetition improved coverage but increased false leads, highlighting the need for harness engineering to balance recall, precision, and triage costs.