How we estimate the risk from prompt injection attacks on AI systems

Google DeepMind presents a threat model for indirect prompt injection against AI agents and an automated red-teaming framework to measure and reduce the risk of sensitive data exfiltration. The framework tests a realistic email scenario and employs three optimization-based attack methods—Actor Critic, Beam Search, and a modified Tree of Attacks with Pruning—to iteratively craft effective malicious prompts. The report emphasizes using rigorous evaluation, monitoring, heuristic defenses, and standard security engineering to strengthen AI systems against these attacks.