Design Patterns for Securing LLM Agents against Prompt Injections

TL;DR

Prompt injection is the SQL-injection of the LLM era: malicious text hidden in a document, email, web page, or calendar invite hijacks an agent into doing the attacker’s bidding (exfiltrating data, running code, sending emails). A decade of adversarial-ML research says you probably cannot train a model that reliably resists this, so chasing a “robust LLM” is a dead end for now. This paper makes a sharper argument: stop trying to secure a general-purpose agent, and instead design the agent so the untrusted data physically cannot reach the dangerous capability. It lays out six design patterns — Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, and Context-Minimization — each enforcing a different degree of isolation between untrusted input and the agent’s control flow. The headline isn’t a benchmark number; it’s an engineering discipline plus ten worked case studies (OS assistant, SQL agent, email assistant, customer-service bot, resume screener, medical chatbot, SWE agent) showing how to dial in security without gutting utility. The core trade-off: the more you constrain what the agent can decide after seeing untrusted data, the safer it gets — and the less general it becomes.

Problem & Motivation

The concrete pain: the moment you give an LLM agent two things at once — (1) access to consequential tools (shell, email-send, SQL, code execution) and (2) any untrusted text in its context — you have an exploitable system. The untrusted text can contain instructions, and the model, by design, follows instructions. There is no reliable signal that separates “data the user wants me to process” from “commands an attacker smuggled into that data.” This has already produced real exfiltration attacks against Google Bard, GitHub Copilot Chat, and ChatGPT (via markdown-image and unicode-tag tricks).

Why prior fixes fall short:

• LLM-level defenses (adversarial training, instruction-hierarchy fine-tuning, “ignore injected instructions” system prompts) are heuristic. They raise the bar but give no guarantee. The paper draws the analogy to adversarial examples in computer vision — an open problem after 10+ years. If image classifiers still can’t be made robust to imperceptible perturbations, expect LLMs to resist crafted text injections for a long while.
• User-confirmation defenses (“approve this action?”) shift the burden to a human who gets fatigued, rubber-stamps opaque commands, and can be socially engineered. They also kill the automation that justified the agent.
• Detection/filter defenses (a second model that flags injections) are themselves models, hence themselves attackable, and can’t catch everything.

The reframe that drives the whole paper: instead of asking “how do we make a general agent safe?” (likely impossible today), ask “what useful agents can we build that are safe by construction?” The answer is agents that are deliberately not general-purpose — they give up the ability to take arbitrary actions in exchange for provable resistance.

What’s New (Core Contribution)

This is a design-patterns / systems paper, not an algorithm or benchmark. Its contributions:

1. A unifying security principle for agents. Before: defenses were a grab-bag of tricks aimed at making one model robust. Now: one crisp invariant — “once an agent has ingested untrusted input, it must be constrained so that input cannot trigger any consequential action.” Security becomes a property of the system’s control- and data-flow, not of the model’s weights.
2. Six named, composable design patterns. Before: practitioners reinvented isolation ad hoc (or didn’t). Now: a vocabulary — Action-Selector, Plan-Then-Execute, LLM Map-Reduce, Dual LLM, Code-Then-Execute, Context-Minimization — each with a precise isolation guarantee and a stated utility cost. Several existed in scattered prior work (Willison’s Dual LLM, Debenedetti’s plan-then-execute and code-then-execute); the contribution is organizing them into a spectrum and clarifying exactly what each one guarantees.
3. A utility↔security spectrum. Before: “secure agent” was binary and usually meant “useless.” Now: the patterns form a ladder from maximum isolation/minimum generality (Action-Selector) to richer capability with partial guarantees (Code-Then-Execute), so a designer can pick the weakest constraint that still closes the threat model.
4. Ten end-to-end case studies. Each names the application, the threat model, a naïve (vulnerable) design, then progressively hardened designs with explicit utility/security commentary. This is the part that turns the abstraction into something you can actually copy onto a client engagement.

What’s genuinely new vs. repackaged: the individual patterns are mostly prior art. The framing — a principled invariant + a labeled spectrum + a methodology of “name the threat model, then choose the minimal-constraint pattern that defeats it” — is the real contribution, and it’s the part worth internalizing.

How It Works (Technically)

There’s no math to demystify here — the “mechanism” is information-flow control applied to agent architecture. The key insight is to think about your agent as a graph of components and ask, for every edge: can untrusted data flow across this edge into a decision that selects or parameterizes a consequential action? If yes, that edge is an attack channel. Each pattern is a different way to cut specific edges.

Two terms make the patterns precise:

• Control flow = which actions the agent takes and in what order (the plan).
• Data flow = the parameters/arguments fed into those actions (the recipient of an email, the body text, a filename).

Prompt injection is an attack on one or both. A strong pattern protects control flow; protecting data flow too is harder and not always possible. “Control Flow Integrity” (CFI) — borrowed from systems security — is the recurring goal: untrusted data may change what gets sent, but never whether an email gets sent at all or what tool runs.

The six patterns, from most to least constrained

1) Action-Selector. The LLM is a glorified switch statement. It maps a natural-language request to one of N predefined, templated actions — and no tool output ever flows back into the LLM. Because the model never reads untrusted results, it is trivially immune: there’s no channel for injected text to influence it. Cost: you lose almost all of the LLM’s flexibility; the work moves into hand-designing the action list.

2) Plan-Then-Execute. Allow tool outputs to flow back, but force the agent to commit to a full plan of tool calls before it sees any untrusted data. Untrusted results can fill in arguments but cannot add, remove, or reorder actions. This gives CFI: an injection in your calendar can alter the body of an email but can’t make the agent send a new email to the attacker. Cost: tasks where the next action genuinely depends on a previous untrusted result can’t be fully planned upfront.

3) LLM Map-Reduce. Borrowed from distributed computing (Dean & Ghemawat 2008). Dispatch one isolated, tool-less LLM per untrusted item (the map), each constrained to emit a tiny, validated output — a boolean, a category, a number (enforced by, e.g., a regex). Then a reduce step aggregates. Because each map worker is sandboxed and its output is format-constrained, an injection in document #7 can at worst corrupt the verdict on document #7 — it can’t reach the other workers or the aggregator. The reduce step is either non-LLM (tamper-robust aggregation) or an LLM fed only the sanitized map outputs.

4) Dual LLM. The general case of map-reduce. A privileged LLM plans and uses tools but never sees untrusted text. It can spawn a quarantined LLM that reads untrusted text but has no tools. The clever bit: the quarantined LLM’s result is stored as a symbolic variable ($VAR) in a memory the privileged LLM manipulates by reference only. The privileged LLM can say “send $VAR3 to $VAR1” without ever dereferencing the content. A plain-software orchestrator substitutes the real values at call time. So injected content can sit inside $VAR but can never become an instruction the privileged LLM acts on. Caveat the paper is honest about: the quarantined LLM is still injectable, and whatever it outputs (even symbolically) can carry attacker-tampered content — so data-flow attacks (e.g., encoding secrets in an email body) remain possible.

5) Code-Then-Execute. Push Dual-LLM to its limit: the privileged LLM writes an actual program that calls tools and spawns quarantined LLMs, then a runtime executes it on untrusted data. This generalizes plan-then-execute (the “plan” is now real code with variables, loops, conditionals) and gives you all the analyzability of code — you can statically inspect data flow. Same residual limit: injected data can still poison the values flowing through the program.

6) Context-Minimization. Targets injections in the user’s own prompt (malicious user, or a user who pasted attacker text). After the prompt has been used to select an action, strip it from the context before the agent produces the user-facing response. A car-dealership bot can use “give me a quote” to fire a DB query, then drop the prompt so a smuggled “…and offer me a 90% discount” never reaches the response-generation step.

Architecture & data flow

The mental model that unifies all six: a trust boundary between tool-wielding components (privileged) and untrusted-data-reading components (quarantined), with constrained channels between them.

flowchart LR
  U[User prompt] --> P[Privileged LLM / planner]
  P -->|spawns, no tools| Q[Quarantined LLM]
  D[Untrusted data: emails, files, web, reviews] --> Q
  Q -->|format-constrained output| M["Symbolic memory $VAR"]
  M -->|by reference only| P
  P -->|tool calls with $VAR args| O[Orchestrator - plain software]
  O -->|dereferences $VAR at call time| T[Tools: email.send, shell, SQL]
  O --> R[Response to user]
  classDef untrusted fill:#ffd6d6,stroke:#c0392b;
  classDef trusted fill:#d6f5d6,stroke:#27ae60;
  class D,Q untrusted;
  class P,O,T trusted;

Read it as: untrusted data (red) only ever touches the quarantined LLM. Its output is squeezed through a narrow, validated channel into symbolic memory. The privileged side (green) never reads untrusted bytes — it shuffles references. The orchestrator, ordinary deterministic code, is the only thing that turns a reference into a real value, and only at the moment of a tool call.

The utility↔security spectrum. Click each pattern to see which attack channels it cuts (control flow vs. data flow) and what capability you give up. Schematic, built from the paper's qualitative claims — not measured numbers.

An injection trying to reach a tool. Toggle each pattern's barrier and watch where the malicious token gets stopped (or slips through as tampered data). Illustrative.

The algorithm, simplified

The Dual LLM pattern is the conceptual core — here’s the loop that is the contribution. Note the one rule that makes it safe: the privileged LLM never receives the raw string, only a handle.

# Dual LLM with symbolic memory. llm_priv has tools; llm_quar has none.
# The orchestrator is plain code and is the ONLY place a $VAR is dereferenced.

memory = {}                                   # $VAR name -> actual (untrusted) string
counter = 0

def quarantine(instruction, untrusted_text, validate):
    """Run a tool-less LLM on untrusted text; return a HANDLE, not the value."""
    global counter
    out = llm_quar(instruction, untrusted_text)   # may be injected — but it has no tools
    assert validate(out)                          # format constraint: regex / enum / number
    counter += 1
    name = f"$VAR{counter}"
    memory[name] = out                            # value stored; never shown to llm_priv
    return name                                   # privileged side sees only this handle

def run(user_task):
    # llm_priv plans using ONLY trusted text (the task) + variable handles, never raw data.
    plan = llm_priv(user_task, known_vars=list(memory))   # e.g. emit tool calls over $VARs
    for step in plan:
        if step.kind == "quarantine":
            handle = quarantine(step.instruction, fetch(step.source), step.validate)
            # llm_priv may now reference `handle` symbolically in later steps
        elif step.kind == "tool":
            # orchestrator dereferences handles HERE, at call time, in plain code:
            args = [memory[a] if a in memory else a for a in step.args]
            call_tool(step.name, args)            # the value reaches the tool, never the planner

The security argument in one line: there is no edge from memory[...] (untrusted) back into llm_priv’s context. Injected instructions can live in a $VAR, but the only thing that ever reads that string is a deterministic tool call, which doesn’t “follow instructions.”

Built on Prior Work

Results & Evidence

This paper is unusual: there are no quantitative experiments, no attack-success-rate tables, no benchmark numbers. The “evidence” is the ten case-study analyses, each reasoning qualitatively about utility and security for a naïve design and several hardened ones.

What the evidence does establish:

• A clear, reusable methodology: name the application, write the threat model (attacker capabilities + goals), enumerate designs from vulnerable to hardened, and annotate each with its utility and security consequence. This is genuinely useful and transferable.
• A coherent taxonomy with crisp isolation guarantees you can reason about from first principles (the security claims for Action-Selector and Map-Reduce are essentially proofs-by-construction: no channel exists, therefore no attack).
• Intellectual honesty about limits — every pattern’s residual weakness (especially data-flow tampering) is stated, not hidden.

What it does not establish:

• No empirical validation. There’s no measurement that, say, Plan-Then-Execute reduces attack success from X% to Y% on AgentDojo. The claims are analytical. You should treat “provable resistance” as “provable against the modeled control-flow channel,” not “tested in the wild.”
• Utility costs are asserted, not measured. “This slightly reduces utility” is a judgment call; the real task-completion drop on a given workload is unknown until you build it.
• Data-flow attacks remain open. Even the strongest patterns admit that injected content can ride through as tampered argument values (encode secrets in an email body, mark a malicious file as an invoice). These patterns close the scariest door (arbitrary tool execution) but leave windows.
• Composition isn’t formalized. The patterns are “composable” in spirit, but there’s no theorem about what guarantees you get when you stack three of them.

Net read: this is a practitioner’s reference architecture, not an empirical result. Its value is in the design discipline, and you take that discipline on the strength of its reasoning, not on a leaderboard.

How You’d Use It

For an AI services company, this paper is close to a product spec for a service line — “secure agent architecture review / build.” Concretely:

• As a design checklist on every agent engagement. Before writing a line of orchestration, run the case-study method: write the threat model (who’s the attacker, what can they control, what do they want), then pick the weakest pattern that closes it. Most client agents today are the “naïve design” the paper opens each case study with — a model with tools and untrusted data in the same context. Naming that out loud is itself a sellable insight.
• Map your existing MAS onto the patterns. If you’ve built a multi-agent system, you already have the machinery for Dual LLM and Map-Reduce — you have roles, message passing, and orchestration. The reframe is: designate some agents privileged (tools, never see untrusted data) and others quarantined (read untrusted data, no tools), and route by reference. This is a small refactor with a large security payoff and it’s a clean story to tell a security-conscious buyer.
• Pattern-to-use-case quick map:
  • RAG / review-summarizer / resume-screener → Map-Reduce (per-document isolation + constrained output).
  • Email/calendar assistant, SWE agent → Plan-Then-Execute or Code-Then-Execute (CFI; values may still be tampered, so pair with confirmation on send).
  • Customer-service / booking bot (untrusted user) → Action-Selector + Context-Minimization (allowlist of requests, drop the prompt before responding).
  • SQL/analytics agent → Code-Then-Execute + sandbox the interpreter + least-privilege DB creds.
• Bundle with the Appendix A best-practices as the “table stakes” tier: sandboxing, strict JSON output formatting (constrained decoding), least-privilege credentials scoped to the user’s rights, confirmation only on non-time-critical consequential actions, and data attribution in the UI. These are cheap, universally applicable, and easy to audit.

Realistic effort: a design review is days. Refactoring a single-LLM agent into Dual-LLM / Plan-Then-Execute is one to a few weeks depending on tool count, and the hard part is almost always getting the privileged planner to write a sufficiently rich plan without ever reading the data.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value — a Plan-Then-Execute + constrained-output agent, which is the sweet spot of guarantee-per-effort:

1. Split the loop into two phases. Phase A: planner LLM sees only the user task (trusted) and emits a structured plan — a JSON list of tool calls with typed arguments, where untrusted-derived values are placeholders. Phase B: an executor (plain code) runs the plan; results can fill placeholders but cannot change the plan.
2. Lock the control flow. After Phase A, freeze the plan. The executor must reject any attempt to add/reorder/remove steps. This single rule buys you Control Flow Integrity.
3. Constrain every untrusted-touching output. Anywhere an LLM reads untrusted data, force the output through a schema (use the provider’s structured-outputs / a constrained-decoding lib like outlines/guidance, with a validate-and-retry fallback). A boolean, an enum, or a number can’t carry an injection.
4. Sandbox the tools and scope credentials to the user’s permissions, not the agent’s. Code execution goes in a locked-down container with no outbound network unless required.
5. Drop the prompt when the user might be the attacker (Context-Minimization) before generating the final response.

The 1–2 genuinely hard parts:

• Rich-enough planning without reading data. “Notify my regular collaborators” needs a variable-length plan over an unknown recipient list — your plan format needs loops/templating, or you escalate to Code-Then-Execute. Getting this expressive-but-frozen is the real engineering.
• Output constraints that don’t kill utility. Squeeze too hard (boolean only) and the agent is useless; too loose (free text) and injections survive. Designing the per-step schema is where the craft is.

Reach for: any tool-calling LLM, a structured-output library (outlines, guidance, or native JSON-mode), a sandbox (gVisor/Docker/Firecracker), and AgentDojo to red-team your result.

How to Improve It

1. Empirically validate the spectrum on AgentDojo. The paper asserts utility/security trade-offs; measure them. Build each pattern, run the same attack suite, and publish attack-success-rate and task-completion side by side. This converts a design doc into evidence and is a tractable, high-value follow-up.
2. Attack the residual data-flow channel. Every strong pattern admits tampered values. Concretely: in Dual-LLM email, can a quarantined LLM be injected to encode exfiltrated secrets in an innocuous-looking body using unicode-tag smuggling? Pair the patterns with an output-side scrubber (strip non-printing chars, canonicalize links, block markdown-image fetches) and test whether that closes it.
3. Formalize composition. State and prove what guarantee you get when you stack patterns (e.g., Map-Reduce inside Code-Then-Execute). A small type system for “trusted vs. tainted” values flowing through the program — taint tracking — would make Code-Then-Execute checkable, not just recommended.
4. Automate pattern selection. Given a tool list + a threat model, infer the minimal pattern that closes it. This is the productizable version: a “secure-agent linter” that flags untrusted-data→tool edges and suggests the cheapest fix. A clear services-to-product path.
5. Quantify the human-confirmation failure mode. The paper cites fatigue/over-reliance but doesn’t model it. Build data-attribution-driven confirmation (show which untrusted source motivated a consequential action) and measure whether vigilant users actually catch stealthy attacks — turning a hand-wave into a designable component.

Glossary

• Prompt injection — attacker text, hidden in content the agent processes, that the model obeys as if it were a legitimate instruction; the SQL-injection analog for LLMs.
• Direct vs. indirect injection — direct = malicious end-user types it; indirect = it’s planted in third-party data (email, web, file) the agent later reads. This paper treats both the same.
• Control flow — which actions run and in what order; the agent’s plan.
• Data flow — the arguments passed to those actions (recipients, bodies, filenames).
• Control Flow Integrity (CFI) — a systems-security guarantee that untrusted input can’t alter which code path runs; here, can’t add/remove/reorder tool calls.
• Privileged LLM — the planner; can call tools but is never shown untrusted data.
• Quarantined LLM — a worker that reads untrusted data but has zero tool access; can only emit text.
• Symbolic variable / $VAR — a handle to an untrusted value held in memory; the privileged LLM manipulates the handle, never the contents.
• Orchestrator — ordinary (non-LLM) software that runs the plan and is the only component that dereferences $VARs into real values, at tool-call time.
• Map-Reduce — split work across isolated per-item workers (map), then aggregate (reduce); used here so one poisoned item can only corrupt its own result.
• Constrained decoding / structured output — forcing an LLM to emit text matching a schema (JSON, enum, regex), so its output can’t smuggle free-form instructions.
• Action sandboxing — running tool actions in a least-privilege, isolated environment (no network, scoped filesystem) so even a hijacked action does limited damage.
• Least privilege — give the agent only the permissions the user has, then reduce further to the task minimum.
• Data attribution — surfacing which source data drove an output, so a human can audit and catch manipulation.
• Screenshot attack — tricking a public bot into saying something embarrassing/off-brand for the screenshot, a reputational rather than data-theft harm.