An introduction to automated LLM red teaming

This article demonstrates automating LLM red teaming using promptfoo against a purposely vulnerable ChainLit chatbot: it explains the testing architecture (target/adversarial/grader models), describes plugins and strategies (security/access-control plugins such as BFLA, BOLA, RBAC and strategies like Iterative Jailbreaks), and shows lab results where the model sometimes performs unauthorized tool actions (e.g., retrieving SharePoint content). The piece emphasizes the probabilistic nature of LLM failures, the need for repeated testing and strategy tuning, and the importance of treating LLMs and agentic systems as attack surfaces requiring systematic security testing.