Security and safety

SQL injection's cousin, except there is no clean equivalent of parameterised queries, so you defend in depth instead.

Prompt injection

the attack was not typed by your user, it was sitting in the web page the agent fetched, and your user is the victim rather than the attacker.

Direct vs indirect injection

someone talked your assistant out of its own rules with a role-play framing and it produced exactly what it was built to refuse.

Jailbreak

someone got your assistant to print its own instructions back at them, which is why nothing confidential belongs in there.

System prompt leakage

the injected instruction told the model to encode the conversation into a URL and fetch it, and the tool you gave it was happy to oblige.

Data exfiltration

the agent held the credentials and someone else supplied the instructions, so your own privileged component did the attacker's work for them.

Confused deputy

you gave the agent a write-capable tool it did not really need, and one bad step later it deleted rows you cannot get back.

Excessive agency

you validate and allowlist what the model returned before your code acts on it, because that output is untrusted input to everything downstream.

Output validation as a security boundary

you run a cheap classifier over the input and over the generated answer before either one reaches a person.

Content moderation

the model declined a completely legitimate request, and your code has to handle that gracefully instead of treating it as a failure.

Refusals

you keep a suite of injection payloads and jailbreak attempts and run it against every release, before someone else runs theirs.

Red teaming

you redact before the text leaves your network, and you know what the provider retains and for how long.

PII handling

you check the hash and the licence before running weights from an account you have never heard of, the same way you would a random binary.

Model and supply chain provenance

you read the list once and it turns into the review checklist you run over anything before it ships.

OWASP Top 10 for LLM applications