Once an application is exposed to users, it faces adversarial prompting, which falls into three main types. Prompt extraction tries to recover your system prompt to replicate or exploit the app. Jailbreaking and prompt injection try to subvert safety features or make the model take unauthorized actions (harmful instructions, dangerous code). Information extraction tries to get the model to reveal sensitive data from its training set or context. Defenses are layered: use adversarial safety benchmarks, run security red-teaming to find weaknesses proactively, be explicit about what the model must not return, repeat the system prompt before and after user input, sandbox generated code, require human approval for impactful actions, define out-of-scope topics, use anomaly detection, and guardrail both inputs and outputs. Security is measured by two metrics in tension — the violation rate (how often attacks succeed) and the false-refusal rate (how often legitimate requests are wrongly refused) — and perfect security with too many false refusals is itself a failure.
Claims
- Prompt attacks are of three main kinds: extraction (of the system prompt), jailbreak/injection, and information extraction (of sensitive data). observation
- Repeating the system prompt before and after user input reminds the model of its constraints. (best practice — context: hardening a user-facing app against injection)
- Run generated code only in isolated environments and require human approval for impactful actions. (best practice — context: agents/apps with code-execution or write capabilities)
- Track both violation rate and false-refusal rate; they trade off. principle
- Perfect security with too many false refusals is a bad user experience, i.e. a failure mode of its own. principle
Related
- Input & Output Guardrails — the mechanism layer these attacks are defended with.
- Post-Training Alignment — the safety features attacks attempt to subvert.
- LLM-as-Judge — used to score toxicity/safety among output guardrails.
- Distillate: AI Engineering in 76 Minutes — Chip Huyen's Book, Speedrun