Prompt injection
What is Prompt injection?
An attack where instructions hidden in user input, or in a document the model reads, override its intended behaviour and make it leak data or take unwanted actions. It tops the OWASP risk list for LLM applications, because models still struggle to tell trusted instructions from untrusted content. Any AI that reads external data is exposed to it.
Why it matters
The uncomfortable fact about prompt injection is that there is no clean fix. Because a model reads instructions and data as one stream of text, it cannot reliably tell a genuine command from one smuggled into a web page or document it was asked to process. Defences reduce the risk without removing it: limit what the model may do, validate its actions, keep untrusted content walled off from trusted instructions, and require a human to approve anything consequential. The danger grows as models gain tools, because an injected instruction that once produced only a bad sentence can now trigger a real action in the world. Any system that reads outside content has to be built on the assumption that some of it is hostile.
In practice
An email assistant meets a message containing hidden text that instructs it to forward the user’s inbox elsewhere. Because the system was built expecting exactly this, the assistant has no access to a forwarding tool and cannot act on the smuggled command. The malicious instruction ends up as a harmless line in a summary. The protection came from limiting what the assistant could do, not from hoping it would recognise the trick.
