Popular AI Systems Still a Work-in-Progress for Security

The report reviews Forescout's evaluation of 50 large language models for offensive-security tasks and finds that while many commercial LLMs can locate simple vulnerabilities and sometimes generate simple exploits, most models fail on complex vulnerability discovery and exploit development; it contrasts general-purpose LLMs with specialized autonomous systems (such as Xbow and Atlantis) that have demonstrated substantial automated vulnerability-finding capabilities, and warns that AI-driven automation will accelerate bug discovery and change penetration-testing workflows while still requiring human oversight to mitigate hallucinations and misuse.