A Survey of Context Engineering for Large Language Models

TL;DR

Everything an LLM does is determined by what you put in its context window, yet the discipline of deciding what goes in there has been a grab-bag of tricks with no shared vocabulary. This paper argues that the real job is no longer writing a clever prompt string — it’s engineering a dynamic, structured information payload assembled from instructions, retrieved knowledge, tools, memory, and live state. It formalizes that as an optimization problem (find the functions that assemble the best context under a length budget), then organizes the entire field into a two-layer taxonomy: foundational components (Retrieve/Generate, Process, Manage) and the system implementations that wire them together (RAG, Memory, Tool-Integrated Reasoning, Multi-Agent). The headline finding from surveying 1400+ papers: models are now great at understanding rich context but still bad at generating long, sophisticated output — a “comprehension–generation asymmetry” that is the field’s central open problem. For anyone building agentic systems, this is less a research paper and more a field map and a design rubric.

Problem & Motivation

The concrete pain: if you build LLM applications for a living, you are already doing context engineering — stuffing a system prompt, bolting on RAG, adding a memory store, defining tool schemas, juggling multi-agent message passing — but you’re doing it ad hoc, with no framework telling you which knobs matter, how they interact, or where the failure modes live. Each sub-field (RAG, memory, tool use, agents) has its own survey, its own jargon, and its own benchmarks, and they’re “predominantly studied in isolation.” That fragmentation means:

• No shared mental model. A RAG engineer and a multi-agent engineer are solving the same underlying problem — what information reaches the model — but can’t see it.
• The math is missing. There were no principled answers to “how much of my context budget should go to retrieved docs vs. memory vs. tool defs?” People guess.
• “Prompt engineering” is the wrong frame. It implies a static string you tune by hand. Real systems assemble context dynamically per request, are stateful, and need systematic debugging — none of which “prompt engineering” captures.

The motivation is also hard-nosed and practical. Self-attention is O(n²) in sequence length, so context is literally expensive — every token you add costs compute, latency, and (on commercial APIs) money. And LLMs hallucinate, are unfaithful to provided context, and are brittle to input phrasing. Better context engineering is the lever that addresses all three: accuracy, cost, and reliability.

What’s New (Core Contribution)

This is a survey, so the novelty is in framing and organization, not a new algorithm. Four genuine contributions:

1. A formal definition of Context Engineering as an optimization problem. Before: “context” = the prompt string, tuned by intuition. Now: context C is the output of an assembly function over typed components, and the goal is to find the functions that build the best context across a whole task distribution under a length constraint (the math is in §How It Works). This reframes a craft as a science with an objective function.

2. A two-layer taxonomy that unifies the field. Before: RAG, memory, tools, and agents were separate literatures. Now: they’re all “System Implementations” built from three shared “Foundational Components” (Retrieve/Generate, Process, Manage). This is the load-bearing contribution — it’s a mental model you can actually hold.

3. Typed context components (cinstr, cknow, ctools, cmem, cstate, cquery). Before: context was undifferentiated text. Now: every piece of context has a type that maps to a specific technical sub-field, so “what’s in my window” becomes a structured budget you can reason about and debug per-component.

4. The comprehension–generation asymmetry as the field’s defining gap. Before: a vague sense that long-form output is hard. Now: a named, evidenced claim across 1400+ papers — context engineering has made models excellent readers of complex context but they remain weak writers of long, coherent, factually-consistent output, and closing that gap is the priority.

Be honest about the hype: the “Bayesian context inference” and “information-theoretic optimality” formulations (eqs. 4–6) are framings, not algorithms anyone runs — no one computes mutual information over answers in practice. They’re useful as a way to think, not a method to deploy. The real value is the taxonomy.

How It Works (Technically)

The intellectual core is the formalization in §3.1. Let me demystify each equation, because the math is simpler than the notation suggests.

Equation 1 — what an LLM is.

$$P_\theta(Y \mid C) = \prod_{t=1}^{T} P_\theta(y_t \mid y_{<t}, C)$$

Plain English: the model generates the output Y one token at a time; each next token’s probability depends on all previous tokens and the context C. What it does operationally: it says the only lever you have at inference time (without retraining θ) is C. That’s the whole reason context engineering exists — C is your control surface.

Equation 2 — context is assembled, not written.

$$C = A(c_1, c_2, \ldots, c_n)$$

Plain English: instead of C being one string you typed, it’s the output of an assembly function A that formats and concatenates a set of typed components. The components are:

• cinstr — system instructions / rules
• cknow — retrieved external knowledge (RAG, knowledge graphs)
• ctools — definitions/signatures of available tools
• cmem — persistent info from prior interactions (memory)
• cstate — live state of the user/world/other agents
• cquery — the user’s immediate request

What it does operationally: this is the entire taxonomy compressed into one line. Every sub-field of the survey is “the set of techniques for producing one of these components well.” When you design an agent, you are literally choosing functions to fill each c.

Equation 3 — the actual optimization.

$$F^* = \arg\max_F ; \mathbb{E}{\tau \sim T}\big[\text{Reward}\big(P\theta(Y \mid C_F(\tau)),, Y^*\tau\big)\big] \quad \text{s.t.}\ |C| \le L{max}$$

Demystified: F is the collection of functions that build your context (the assembler A, your Retrieve, your Select, etc.). τ is a task drawn from the distribution of tasks T your system faces. For each task, F produces a context C_F(τ), the model answers, and you score that answer against the ideal Y* with a Reward function. The objective: find the set of context-building functions that maximizes expected reward across all your tasks, subject to the context-length budget L_max.

The key shift versus prompt engineering (their Table 1): prompt engineering optimizes over a string space for one prompt; context engineering optimizes over a function space — your retrieval logic, your memory policy, your formatting — at the system level. This is why it’s an engineering discipline: you’re tuning a pipeline, not a sentence.

Equations 4–6 — the theoretical lens (read as intuition, not code).

• Eq. 4 says retrieval should maximize the mutual information between the retrieved knowledge and the correct answer, given the query — i.e., don’t fetch what’s merely similar to the query, fetch what’s informative for answering it. (In practice this is the difference between naive semantic search and answer-aware/reranked retrieval.)
• Eqs. 5–6 wrap the whole thing in Bayesian inference: treat the ideal context as a posterior P(C | query, history, world) and pick the context that maximizes expected reward. The practical takeaway: handle context adaptively — update what you include as you learn more across multi-step reasoning. No one literally integrates eq. 6; it’s a principled justification for adaptive/iterative retrieval and memory.

Architecture & data flow

flowchart TB
  Q[cquery: user request] --> A
  I[cinstr: system rules] --> A
  K[cknow: retrieved knowledge<br/>RAG / KG] --> A
  T[ctools: tool definitions] --> A
  M[cmem: persistent memory] --> A
  S[cstate: world / agent state] --> A
  A[Assembly function A<br/>format + select + concat<br/>under length budget Lmax] --> C[Context C]
  C --> LLM[LLM Pθ]
  LLM --> Y[Output Y]
  Y -. tool call .-> EXT[Tools / Environment]
  EXT -. results .-> M
  Y -. write .-> M
  Y -. update .-> S
  subgraph Components [Foundational Components]
    direction LR
    GEN[Retrieve &amp; Generate] --- PROC[Process] --- MAN[Manage]
  end
  Components -.governs.-> A

The loop you’d actually write: gather typed components → assemble (select + format + fit to budget) → call model → if the model emits a tool call, execute it and feed results back into cknow/cstate → write durable facts to cmem → repeat. The “foundational components” (Retrieve/Generate, Process, Manage) are the operations you run on these pieces; the “system implementations” (RAG, Memory, Tools, Multi-Agent) are named recipes that combine them.

Interactive context-budget allocator (schematic): drag the sliders to split a fixed L_max token budget across the six context components and watch the "fit" indicator — it illustrates the core constraint in Eq. 3, that context engineering is allocation under a hard budget, not infinite stuffing.

The taxonomy as a build checklist

The survey’s spine, reorganized as the decisions you make when building:

Layer 1 — Foundational Components (operations on context):

1. Context Retrieval & Generation — prompt engineering (CoT, few-shot), external knowledge retrieval (RAG, KG), and dynamic context assembly (the A function itself).
2. Context Processing — long-context handling (attention tricks, position interpolation), self-refinement (the model critiques and improves its own context/output), multimodal context, and structured/relational context (graphs, tables).
3. Context Management — memory hierarchies (working vs. long-term), context compression (squeeze more signal per token), and managing the hard L_max constraint.

Layer 2 — System Implementations (named recipes):

1. RAG — modular → agentic (the retriever is itself an agent that decides when/what to fetch) → graph-enhanced (GraphRAG, retrieving over a knowledge graph).
2. Memory Systems — architectures for persistent state (MemGPT, Mem0, A-MEM) and memory-enhanced agents.
3. Tool-Integrated Reasoning — function calling, interleaving reasoning with tool calls (ReAct, ToRA), and agent–environment interaction.
4. Multi-Agent Systems — communication protocols (MCP, A2A, ACP), orchestration, and coordination strategies.

The algorithm, simplified

There’s no single algorithm in a survey, so here is the context-engineering loop the formalization implies — the thing the whole taxonomy is describing. This is the core idea you’d actually type:

# Context engineering = assemble typed components under a length budget, per request.
# Stubs: retrieve()->list[str], recall()->list[str], llm(ctx)->str, run_tool()->str

def answer(query, agent_state, memory, tools, L_max):
    # 1. RETRIEVE & GENERATE: gather each typed component (Eq. 2: C = A(c1..cn))
    c_instr = SYSTEM_RULES
    c_query = query
    c_tools = format_tool_signatures(tools)          # ctools
    c_mem   = recall(memory, query)                  # cmem: prior-interaction facts
    c_state = agent_state.summary()                  # cstate: world / other agents
    c_know  = retrieve(query, rerank_by="answer_informativeness")  # Eq. 4, not raw similarity

    # 2. PROCESS: refine/compress each piece so it earns its tokens
    c_know = compress(c_know)                         # Context Management: more signal/token

    # 3. MANAGE: assembly function A — select + format + FIT to budget (Eq. 3 constraint)
    pieces = prioritize([c_instr, c_query, c_tools, c_mem, c_state, c_know])
    context = fit_to_budget(pieces, L_max)            # drop/summarize lowest-value pieces

    # 4. CALL MODEL (Eq. 1: P(Y|C))
    out = llm(context)

    # 5. ACT + UPDATE STATE (closes the loop; this is what makes it stateful, not a prompt)
    if is_tool_call(out):
        result = run_tool(out, tools)
        agent_state.record(result)                   # feeds back into c_state / c_know next turn
        return answer(query, agent_state, memory, tools, L_max)  # iterate (Bayesian update, Eq. 5)

    memory.write(extract_durable_facts(query, out))  # persist for future cmem
    return out

The line that is the contribution: fit_to_budget(prioritize([...]), L_max). Prompt engineering has no such line — it has a string. Context engineering treats your window as a scarce resource and makes per-request allocation decisions across typed components.

Built on Prior Work

The lineage point: nothing here is a new technique. The delta is showing that all of these are instances of one optimization problem and giving them a shared coordinate system. For a practitioner that’s genuinely useful — it turns “I should add memory” into “I’m choosing a function to produce cmem and budgeting tokens for it against cknow.”

Results & Evidence

This is a survey, so “results” = the synthesis, not an experiment. What the evidence supports:

• Breadth. 1400+ papers organized into a coherent, non-overlapping taxonomy. The taxonomy holds up — the typed-component model genuinely maps onto the sub-fields without forcing.
• The asymmetry claim. Across the surveyed work, models + good context engineering excel at comprehension benchmarks (long-doc QA, retrieval-heavy tasks) but degrade sharply on long-form generation (coherence over thousands of tokens, factual consistency, multi-step planning). They cite this gap repeatedly across §7. This is the most actionable finding: it tells you where the unsolved money is.
• Cited improvement numbers (e.g., 18× text-navigation accuracy, 94% success rates from RAG/structured prompting) — but treat these as illustrative cherry-picks from individual papers, not head-to-head comparisons. A survey can’t control for baselines across 1400 sources.

What the evidence does not establish:

• No empirical ranking. It won’t tell you “agentic RAG beats modular RAG by X% on your task.” There are no controlled experiments here — it’s a map, not a benchmark.
• The math is unvalidated as method. Eqs. 4–6 are not shown to improve any system; they’re a conceptual frame. Don’t expect a Bayesian context optimizer to fall out of this.
• Recency bias / fast decay. A 2025 survey of a field moving this fast is a snapshot. The taxonomy will age better than the specific systems named.

How You’d Use It

For an AI services company, this is a client-facing diagnostic and a build rubric, not a paper to implement.

• As an audit framework. Walk a client’s existing LLM app through the six components: Are you doing cknow (retrieval)? Is it answer-aware or naive similarity? Do you have cmem at all? Who manages L_max — or are you just truncating? Most production apps are missing 2–3 components entirely. That’s a gap analysis and a statement of work in one pass.
• As a vocabulary with clients and your own team. “We need to improve your cmem policy and compress cknow to free budget for tool defs” is a precise, scoped conversation. It turns vibes into line items.
• As a build-vs-buy lens. The taxonomy tells you which layers are commoditized (basic RAG, function calling — buy) vs. where the moat is (good cmem policies, answer-aware retrieval, multi-agent cstate orchestration — build/differentiate).
• To pick your bet. The comprehension–generation asymmetry says: if you want defensible value, work on the generation side — long-form coherence, planning, verification loops — because that’s where everyone is weak. Comprehension/RAG is increasingly table stakes.

Where it slots into your MAS work: cstate is exactly the agent-coordination pain you’ve already hit. The survey’s framing — that inter-agent information is just another typed context component that must be assembled and budgeted — is a clean way to reason about why multi-agent systems blow their context budgets and drift.

Build Your Own (Minimal Recipe)

The smallest thing that captures 80% of the value isn’t reimplementing the survey — it’s building a context assembler with explicit budgeting, which most teams skip.

1. Define typed components as a schema. A dict/dataclass with the six keys (instr, query, tools, mem, state, know). Everything that goes to the model passes through it. (Half a day.)
2. A fit_to_budget(pieces, L_max) function. Count tokens (tiktoken), assign each component a priority + a max share, and when over budget, summarize/drop lowest-priority pieces. This single function is the conceptual heart and almost nobody writes it. (1 day.)
3. Answer-aware retrieval for cknow. Start with vector search, then add a reranker (a cross-encoder or an LLM judge scoring “does this help answer the query?”). This approximates Eq. 4 cheaply. (1–2 days.)
4. A minimal cmem. Write extracted facts to a store keyed by user/session; recall() fetches top-k relevant on each turn. (1 day; use Mem0 or a sqlite + embeddings table.)
5. The loop. Wire steps into the answer() function above, with tool calls feeding back into state.

The two genuinely hard parts: (a) fit_to_budget prioritization — deciding what to drop when over budget is where quality is won or lost, and it’s task-specific; (b) memory policy — what to persist and when to recall without polluting the window. Reach for: tiktoken (counting), a reranker model (BGE-reranker or an LLM call), Mem0/LlamaIndex (memory + retrieval scaffolding), and LangGraph if you want the loop as a graph.

How to Improve It

Limitations as leverage — concrete, testable directions:

1. Turn the budget allocator into a learned policy. The survey leaves fit_to_budget as hand-tuned. Train a small policy (even a bandit) that learns per-task-type how to split L_max across components, scored by downstream reward (Eq. 3 made real). Testable: does learned allocation beat fixed shares on a multi-task eval?
2. Operationalize Eq. 4. Build a reranker explicitly trained to estimate answer-informativeness (mutual-information proxy) rather than query similarity, using QA pairs. Measure retrieval precision on answer-bearing chunks vs. a standard embedder.
3. Attack the comprehension–generation gap directly. The survey names it but doesn’t solve it. A verifier-in-the-loop generation harness (draft → critique against cknow → revise) targeting long-form factual consistency is an obvious, fundable bet. Test on long-form generation benchmarks for factual-consistency-per-1000-tokens.
4. Cross-component interference study. The taxonomy assumes components compose cleanly, but they compete for attention. Empirically measure: does adding cmem hurt cknow utilization at fixed budget? This is the “compositional understanding” gap §7.1.1 flags — and it’s a paper.
5. Cost-aware objective. Eq. 3 constrains only L_max, but tokens cost money and latency. Add an explicit cost term and optimize reward-per-dollar. Directly relevant to anyone deploying commercially.

Glossary

• Context Engineering — the discipline of systematically assembling and optimizing the full information payload (C) an LLM sees, beyond just the prompt string.
• Context window / L_max — the hard limit on how many tokens fit in C; the binding constraint in the optimization.
• Assembly function A — the pipeline that selects, formats, and concatenates typed components into the final context.
• Typed components (cinstr etc.) — the six categories of context: instructions, knowledge, tools, memory, state, query.
• RAG (Retrieval-Augmented Generation) — fetch external documents at inference and inject them as cknow.
• Agentic RAG — RAG where an agent decides when and what to retrieve, possibly iteratively, instead of a fixed one-shot fetch.
• GraphRAG — retrieval over a knowledge graph so the model gets relational structure, not just flat passages.
• Tool-Integrated Reasoning — interleaving model reasoning with tool/function calls (e.g., ReAct, ToRA) so the model can act and observe.
• Mutual information I(Y*; cknow | cquery) — a measure of how much the retrieved knowledge tells you about the correct answer; the ideal retrieval target.
• Bayesian context inference — framing context selection as inferring the most-likely-useful context given query, history, and world state, updated as you learn more.
• Self-refinement — the model critiques and revises its own output/context across iterations.
• Context compression — reducing tokens while preserving task-relevant signal, to fit more under L_max.
• Comprehension–generation asymmetry — the survey’s central finding: models understand complex context far better than they can generate long, coherent, factual output.
• MCP / A2A / ACP — emerging protocols for tool access and agent-to-agent communication (the plumbing of cstate in multi-agent systems).
• State space models (e.g., Mamba) — non-transformer architectures with linear (vs. quadratic) scaling in sequence length, a candidate for cheaper long contexts.