NDSS 2025 – A Comparative Evaluation Of Large Language Models In Vulnerability Detection

This NDSS conference paper evaluates multiple large language models (LLaMA-2, CodeLLaMA, LLaMA-3, Mistral, Mixtral, Gemma, CodeGemma, Phi-2, Phi-3, GPT-4) for automated vulnerability detection, primarily in Java with additional C/C++ tests. The study measures runtime and detection accuracy in zero-shot and few-shot settings, examines effects of model parameters (including quantization and context window size), and finds that while some models (e.g., Gemma, LLaMA-2) can perform well, performance varies widely by model, language, and configuration, with limitations remaining for generalized, reliable vulnerability identification.