A Survey of Context Engineering for Large Language Models

TL;DR

LLM quality is overwhelmingly decided by the context you give the model, not by clever wording of a single prompt. The authors promote “Context Engineering” from a folk art (“prompt engineering”) into a formal discipline: a context is no longer a static string, it is a dynamically assembled set of components (instructions, retrieved knowledge, tool definitions, memory, world state, the query), and the engineering problem is to find the functions that assemble the best context for each task under a token-budget constraint. They organize the entire field into three foundational components (Retrieval/Generation, Processing, Management) and four system implementations that combine them (RAG, Memory Systems, Tool-Integrated Reasoning, Multi-Agent Systems). The headline empirical finding from surveying the literature: models are now great at understanding huge, complex contexts but remarkably bad at generating equally long, coherent, factual outputs — a “comprehension-generation asymmetry” that the authors flag as the field’s defining open problem.

Problem & Motivation

The concrete pain: anyone building real LLM systems has discovered that “prompt engineering” — tweaking one string until it works — does not scale. The moment you add a retriever, a memory store, a set of tools, and a few cooperating agents, you are no longer writing a prompt; you are building an information supply chain, and there is no shared vocabulary, no formal objective, and no map of which techniques solve which sub-problem.

Why prior framing falls short:

• Prompt engineering treats context as a monolithic static string. That view is stateless, brittle as length grows, and offers no principled way to decide what to include, what to drop, or how to order it.
• The literature is fragmented. RAG people, long-context people, memory people, and multi-agent people publish in separate silos with overlapping ideas and no unifying objective. There was no single framework saying “these are all instances of the same optimization problem.”
• Real costs are systemic, not lexical. Self-attention is O(n²) in sequence length, so context bloat directly drives latency and token-billing cost. Hallucination and “lost-in-the-middle” failures are context-assembly failures, not wording failures. None of that is addressable by editing a prompt string.

The survey’s motivating claim: if you formalize context as a structured, optimizable object, every one of these problems becomes a tractable engineering target with measurable components you can evaluate and debug independently.

What’s New (Core Contribution)

This is a survey, so the novelty is conceptual organization rather than a new algorithm. Four genuine contributions:

1. A formal definition of Context Engineering as an optimization problem.

   • Before: “prompt engineering” = search over the space of strings to maximize P(Y | prompt).
   • Now: find the set of context-generating functions F (assemble, retrieve, select, format, compress…) that maximizes expected task reward across a task distribution, subject to |C| ≤ L_max. The thing you optimize is no longer text — it is the pipeline that produces text.

2. A component decomposition of “context.” Context C = A(c_instr, c_know, c_tools, c_mem, c_state, c_query) — six typed slots that map one-to-one onto the technical subfields of the survey. This is the genuinely useful reframe: it turns “what goes in the prompt?” into a typed schema with an owner for each slot.

3. A two-layer taxonomy: Components → Implementations. Three foundational components (Retrieval/Generation, Processing, Management) are the primitives; four implementations (RAG, Memory, Tool-Integrated Reasoning, Multi-Agent) are the systems that wire the primitives together. This Lego-brick framing is the survey’s backbone.

4. An empirically-grounded research gap: the comprehension–generation asymmetry. Synthesizing 1400+ papers, they argue current models, when well-fed, understand arbitrarily complex contexts but cannot generate equally long, coherent, factual long-form output. This is presented as the field’s top priority, and it is the most actionable single takeaway.

What is repackaging, honestly: “context engineering” as a buzzword was already circulating in 2024-2025 practitioner circles (Karpathy, LangChain, etc.). The survey’s value-add is the formalism plus the taxonomy plus the citation map, not the coinage.

How It Works (Technically)

The “mechanism” of a survey is its framework. Let me demystify the math first, because the formal definition is the part worth internalizing — it changes how you architect systems.

Equation 1 — the baseline. An autoregressive LLM with weights θ generates output Y from context C by maximizing P_θ(Y|C) = Π_t P_θ(y_t | y_<t, C). Plain English: the model produces one token at a time, each conditioned on everything before it and on the context. The whole point of the survey is that C — not θ — is the lever you control at inference.

Equation 2 — context as assembly. Instead of C = prompt, write C = A(c_1, …, c_n). A is an assembly function: it sources, filters, formats, and concatenates components. Operationally, A is your prompt-construction code — the function that decides “system instructions first, then 5 retrieved chunks, then tool schemas, then the last 3 conversation turns, then the query.” The six typed slots:

• c_instr — system rules and persona.
• c_know — retrieved external knowledge (RAG, knowledge graphs).
• c_tools — tool/function signatures available to the model.
• c_mem — persisted information from prior turns/sessions.
• c_state — live state of the user, world, or other agents.
• c_query — the immediate user request.

Equation 3 — the objective. F* = argmax_F E_{τ∼T}[Reward(P_θ(Y|C_F(τ)), Y*_τ)] subject to |C| ≤ L_max. Translation: over a distribution of tasks T, find the set of context-building functions F that maximize expected reward (how good the output is vs. the ideal Y*), under the hard ceiling of the context window. The shift from optimizing a string for one task to optimizing a pipeline across many tasks is the entire conceptual jump. In RL terms, Reward is your scalar feedback signal; F is effectively a policy over how to build context.

Equation 4 — retrieval as information theory. Retrieve* = argmax I(Y*; c_know | c_query). The best retriever maximizes the mutual information between the retrieved knowledge and the correct answer, given the query. The practical lesson: semantic similarity to the query is the wrong target. You want chunks that are maximally informative about the answer — which is why pure cosine-similarity retrieval underperforms and reranking/query-rewriting help.

Equations 5-6 — Bayesian context inference. Rather than deterministically building one context, treat the optimal context as a posterior: P(C | c_query, History, World) ∝ P(c_query | C) · P(C | History, World), then pick C* to maximize expected reward integrated over possible answers. The takeaway for builders: this licenses adaptive retrieval — update your belief about what context is needed as a multi-step task unfolds, instead of retrieving once up front. That is exactly what agentic/iterative RAG does in practice.

The two-layer taxonomy (the heart of the survey).

Foundational Components:

• Context Retrieval & Generation — prompt engineering (CoT, few-shot, role prompting), external knowledge retrieval (RAG, KG lookup), and dynamic context assembly (the A function).
• Context Processing — long-sequence handling (position interpolation, attention tricks for ultra-long inputs), self-refinement (the model critiques and revises its own output, e.g. Self-Refine / Reflexion-style loops), multimodal context, and structured/relational context (graphs, tables).
• Context Management — fundamental constraints (the O(n²) wall, the finite window), memory hierarchies and storage, and context compression (token-level pruning, summarization to fit budget).

System Implementations (components wired together):

• RAG — modular pipelines, agentic RAG (the model decides when/what to retrieve), graph-enhanced RAG (GraphRAG).
• Memory Systems — architectures for persistent state across sessions; memory-enhanced agents.
• Tool-Integrated Reasoning — function calling, interleaving reasoning with tool calls (ToRA, Toolformer lineage), agent-environment interaction.
• Multi-Agent Systems — communication protocols, orchestration, coordination strategies.

Architecture & data flow

flowchart TB
  subgraph Components["Foundational Components (primitives)"]
    RG["Retrieval & Generation<br/>prompts, RAG, assembly A()"]
    PR["Processing<br/>long-context, self-refine, structured"]
    MG["Management<br/>memory hierarchy, compression, budget"]
  end
  subgraph Slots["Typed context C = A(...)"]
    CI[c_instr] --- CK[c_know] --- CT[c_tools] --- CM[c_mem] --- CS[c_state] --- CQ[c_query]
  end
  subgraph Systems["System Implementations (compositions)"]
    RAG[RAG: modular / agentic / graph]
    MEM[Memory Systems]
    TIR[Tool-Integrated Reasoning]
    MAS[Multi-Agent Systems]
  end
  Components --> Slots
  Slots -->|assembled context| LLM["LLM  P_theta(Y|C)"]
  Systems --> Slots
  LLM --> OUT["Output Y"]
  OUT -->|reward / feedback| Components

Schematic of the context-window budget problem (Eq. 3's |C| ≤ L_max constraint). Drag the sliders mentally: as you add retrieved chunks, memory, and tool schemas, the fixed window fills up and the "useful signal" must compete with overhead. This illustrates why compression and selection are first-class engineering concerns, not afterthoughts.

The algorithm, simplified

A survey has no single algorithm, but the assembly function A is the one idea every reader should be able to write. Here is the core context-engineering loop the formalism implies — a budget-aware, mutual-information-flavored assembler:

# Build the optimal context C = A(c_instr, c_know, c_tools, c_mem, c_state, c_query)
# under the hard constraint |C| <= L_max. This IS context engineering in code.

def assemble_context(query, store, memory, tools, L_max):
    c_instr = SYSTEM_RULES                      # c_instr: fixed persona + rules
    c_query = query                              # c_query: the immediate request

    # c_know: retrieve, then rerank by informativeness, not raw similarity (Eq. 4)
    candidates = store.search(query, k=20)       # cheap recall
    c_know = rerank_by_mutual_info(query, candidates)  # keep chunks that move the answer

    c_mem   = memory.relevant(query, k=5)        # c_mem: persistent prior-turn facts
    c_tools = select_tools(query, tools)         # c_tools: only signatures we might need
    c_state = current_world_state()              # c_state: live env / agent state

    # Greedy budget fill: order by value-per-token, compress to fit (Context Management)
    parts = priority_order([c_instr, c_query, c_know, c_mem, c_tools, c_state])
    context, used = [], 0
    for p in parts:
        if used + tok(p) > L_max:
            p = compress(p, budget=L_max - used)  # summarize/prune to fit, don't drop blindly
        context.append(p); used += tok(p)
    return format_for_model(context)              # A = Concat o (Format_1..Format_n)

The lessons baked in: retrieve wide then rerank for informativeness; treat the window as a budget you greedily fill by value-per-token; and when something doesn’t fit, compress it rather than silently truncate. Everything in the survey’s “Context Management” section is elaboration on that for loop.

Built on Prior Work

The lineage move that matters: the survey absorbs five previously-separate subfields into slots of one objective function, so you can reason about trade-offs between them (more retrieval vs. more memory vs. bigger tool set) within a single token budget.

Results & Evidence

This is a literature synthesis, not an experiment, so “results” means claims distilled from cited work and the survey’s own structural argument.

What the evidence supports:

• Context engineering yields large, documented gains in specific settings: cited improvements include an 18× boost in text-navigation accuracy, 94% success rates in some pipelines, +9.90% BLEU-4 on code summarization from few-shot selection, and +175.96% exact-match on bug fixing. These are real, paper-backed numbers — but each comes from a different paper and a different benchmark.
• The taxonomy is comprehensive: 1400+ papers is a credible coverage claim, and the Components→Implementations split is internally consistent.
• The comprehension-generation asymmetry is supported by multiple cited evaluations (e.g., agents that parse complex multimodal inputs but fail at extended coherent generation; degradation on long-form output tasks).

What it does NOT establish:

• No head-to-head comparison. Because numbers are pulled from heterogeneous sources, you cannot conclude “agentic RAG > modular RAG by X%.” The survey maps the field; it does not benchmark it.
• Cherry-picked highs. Figures like “18×” and “175.96%” are best-case results on favorable tasks; the survey reports them as evidence of potential, not typical lift. Treat them as ceilings, not expectations.
• The formalism is descriptive, not predictive. Equations 3-6 are a clean way to think; the survey itself notes (in Future Directions) that there are no theoretical bounds, no scaling laws, and no way to predict the optimal context composition. The math frames the problem; it does not solve it.
• Survey recency / authorship lens. Heavily weighted toward 2023-2025 work; some “categories” are thin and may not survive as the field consolidates.

Net: trust the framework and the asymmetry finding; be skeptical of any single cited percentage as a planning number.

How You’d Use It

For someone running an AI services company and building agentic/multi-agent systems, this paper is best used as an architecture checklist and a sales narrative, not as a thing to implement.

• Audit existing client systems against the six slots. Most underperforming RAG/agent deployments are failing one specific slot: weak c_know (bad retrieval), bloated c_instr, or no c_mem. The typed decomposition gives you a fast diagnostic: “your agent has no c_state sharing, that’s why your multi-agent handoffs lose context.”
• Reframe “prompt tuning” engagements as “context engineering.” This is a genuine commercial upgrade: instead of selling prompt tweaks (low margin, looks like a commodity), you sell information-pipeline design — retrieval quality, memory architecture, budget management, evaluation per component. The survey’s formalism is ready-made client-facing language.
• Make the token budget a line item. Eq. 3’s |C| ≤ L_max constraint maps directly to cost. Selling “context compression” / “context selection” as a cost-optimization service (cut tokens 40% at equal quality) is a concrete, measurable offering, especially under per-token billing.
• Use the comprehension-generation gap to set client expectations. When a client wants the agent to write a 20-page report autonomously, this paper is your evidence that long-form generation is the weak link — budget for human-in-the-loop or sectioned generation rather than promising one-shot long output.
• Multi-agent design: treat inter-agent communication as c_state engineering. The survey’s coordination section reinforces what you already learned building a MAS — the bottleneck is shared context, not model intelligence.

Build Your Own (Minimal Recipe)

You don’t “build a survey,” but you can build the context engineering layer it formalizes — a reusable assembler + evaluator that 80% captures the value:

1. A typed context assembler (A). A single function that takes the six slots and a budget, and returns the formatted prompt. Start with the pseudocode above. Hardest part: deciding priority order and compression policy per task type — this is where the real engineering lives.
2. A retrieval slot with reranking. Vector store (Chroma/pgvector/LanceDB) for recall, then a cross-encoder reranker or an LLM-as-reranker to approximate the mutual-information objective. Don’t ship cosine-only retrieval.
3. A memory slot. Start dead simple: a per-user store of summarized prior turns + extracted facts, retrieved by the same retriever. You can graduate to a MemGPT-style hierarchy later.
4. A budget/compression module. Token-count every slot; when over budget, summarize the lowest-priority slot rather than truncate. LLMLingua-style prompt compression is a drop-in.
5. Per-component evaluation. The survey’s biggest practical gift: evaluate each slot independently. Retrieval recall@k for c_know, fact-recall for c_mem, end-to-end reward for the whole. This is what makes the system debuggable.

Reach for: an orchestration library you already know (LangGraph, your own MAS spine), a vector DB, a reranker model, and an LLM-as-judge for the reward signal. The two genuinely hard parts: (a) the compression/selection policy under budget, and (b) building a task-representative eval set so Reward means something.

How to Improve It

Limitations the survey itself names are the openings:

1. Close the comprehension-generation gap with structured long-form generation. The survey flags this as #1 but offers no method. Concrete attack: a plan-then-fill agent that generates a long output section-by-section, each section a separate context-engineering call with the running document as c_state. Testable against long-form coherence benchmarks.
2. Make A learned, not hand-coded. The objective F* = argmax_F Reward is begging for RL. Train a small policy (or bandit) to decide slot priority and how many chunks to retrieve per query, using end-task reward — i.e., GRPO/PPO over the context-assembly decisions, not the generation. Nobody has cleanly productized “RL over context construction.”
3. Operationalize the mutual-information retriever (Eq. 4). Mutual information is invoked but never computed. Approximate I(Y*; c_know | c_query) with an LLM scoring “how much does this chunk change my answer?” and rerank on that — a concrete, buildable improvement over similarity reranking.
4. Component-interaction theory. The survey admits there’s no model of how slots interfere (e.g., does more c_know crowd out c_mem?). Empirically map the trade-off surface for a fixed budget; that alone would be a publishable, and commercially useful, result.
5. A “living” eval harness. They call for benchmarks that co-evolve with capability. Build a continuously-refreshed, per-component eval suite as a product — it’s the missing measurement layer the whole discipline needs.

Glossary

• Context Engineering — the discipline of systematically assembling and optimizing the information fed to an LLM at inference, treated as a formal optimization problem rather than prompt wording.
• Assembly function (A) — the code that sources, filters, formats, and concatenates context components into the final prompt; C = A(c_1, …, c_n).
• Typed context slots — the six categories of context: instructions, knowledge, tools, memory, state, query.
• Mutual information I(Y; c_know | c_query)* — a measure of how much retrieved knowledge reduces uncertainty about the correct answer; the ideal retrieval target (vs. mere similarity).
• Bayesian context inference — treating the best context as a posterior to be inferred and updated as a task unfolds, enabling adaptive/iterative retrieval.
• RAG (Retrieval-Augmented Generation) — injecting externally retrieved knowledge into the context before generation; modular → agentic → graph-enhanced variants.
• Agentic RAG — the model decides when and what to retrieve mid-reasoning, rather than retrieving once up front.
• Tool-Integrated Reasoning — interleaving model reasoning with tool/function calls so external computation becomes part of the context.
• Context compression — summarizing or pruning content (token-level or semantic) to fit the window budget without losing critical signal.
• Comprehension-generation asymmetry — the survey’s key finding that LLMs understand complex contexts far better than they can generate equally complex long-form output.
• O(n²) attention — self-attention cost grows quadratically with sequence length, the core constraint driving context-management techniques.
• State Space Models (Mamba) — an architecture with linear (not quadratic) scaling in sequence length, a candidate for efficient long-context processing.
• L_max — the model’s hard context-window limit; the binding constraint in the context-engineering optimization.