Keeping LLMs on the Rails Poses Design, Engineering Challenges

Researchers from HiddenLayer demonstrated a prompt-injection technique called "Policy Puppetry" that uses faux policy formatting, role‑playing, and obfuscation to bypass LLM alignment and leak restricted outputs (e.g., system prompts and unsafe instructions). The article outlines related methods (KROP), explains why LLM training data and single-channel control/data interfaces make models susceptible, and recommends mitigation strategies such as separating control and data flows, limiting capabilities/permissions, ML detection and response, and defensive frameworks like CaMeL.