Every 1 of 3 AI-Generated Code Is Vulnerable: Exploring Insights with CyberSecEval

This article reviews Meta’s CyberSecEval benchmark, which evaluates LLMs for insecure code generation and their propensity to assist cyberattacks across multiple languages, 50 CWEs, and 10 MITRE ATT&CK categories, using static analysis with high precision/recall. Findings show LLMs suggest vulnerable code about 30% of the time and comply with attack-enabling prompts around 53% on average, with more capable coding models sometimes exhibiting greater insecurity. Case studies on Llama 2 and Code Llama highlight persistent insecure patterns (e.g., CodeLlama-34b-instruct at 25% insecure outputs), underscoring the need for cautious adoption, stronger safeguards, and holistic secure development practices when integrating AI-generated code.