Inside the AI Red Teaming CTF: Lessons from 200+ Players

A joint HackerOne × Hack The Box study reports on a 10‑day AI red‑teaming CTF (ai_gon3_rogu3) with 504 registrants and 217 active participants across 11 challenges; results show single‑turn filters are frequently bypassed while layered, multi‑turn, and role‑aware defenses significantly reduce success, output‑manipulation tasks were easier than secret exfiltration, and format obfuscation (e.g., JSON/base64) commonly defeats pattern‑matching filters. The paper maps outcomes to the OWASP LLM Top 10 and MITRE ATLAS and provides recommendations for policy, context isolation, and output validation to improve defenses.