Context Engineering 2.0: The Context of Context Engineering

TL;DR

LLM agents live or die by what’s in their context window, but “context engineering” is usually treated as a grab-bag of prompt tricks invented yesterday. This paper reframes it as a single, decades-old problem: reducing the entropy of human intent so a machine of a given intelligence level can understand it. From that lens it builds a four-stage model (Era 1.0 sensor/rule systems → 2.0 today’s LLM agents → 3.0 human-level → 4.0 superhuman) and a practical taxonomy of design choices across three axes — how you collect & store context, how you manage it (compress, organize, abstract), and how you use it (share across agents, select the right subset, infer unspoken needs). It’s a survey/position paper, not an experiment, so the value is the map and the vocabulary, not new benchmark numbers. For anyone building agents, it’s the most complete checklist of “the decisions you didn’t know you were making about context” that currently exists.

Problem & Motivation

Here’s the concrete pain. You build an agent. It works in the demo. Then a real task runs for 200 turns, the context window fills with tool dumps and stale dialogue, the model starts ignoring the instructions from turn 3, retrieval keeps surfacing near-duplicate junk, and two sub-agents disagree because they never shared what they learned. Every one of those failures is a context engineering failure — but the field has no shared vocabulary for them, so each team reinvents ad-hoc fixes (summarize-every-N-turns, a todo.md file, a vector store) without knowing they’re solving instances of the same problem.

The paper’s diagnosis: the community defines “context” too narrowly (dialogue history + system prompt) and treats the discipline as brand new. Both are wrong. The narrow definition throws away 20+ years of HCI / ubiquitous-computing research that already worked out how to model “the situation of an entity,” and the “it’s new” framing means we keep relearning lessons. The deeper framing they offer is entropy reduction: humans communicate by relying on the listener to fill gaps (shared knowledge, tone, situation). Machines can’t fill gaps — at least not yet — so someone has to pre-digest high-entropy human intent into a low-entropy form the machine can consume. That preprocessing effort is the whole game, and the amount of effort required shrinks as machine intelligence grows.

What’s New (Core Contribution)

This is a position + survey paper. The novelty is conceptual scaffolding, not an algorithm.

• A formal, era-agnostic definition of context engineering. Before: “context = the prompt,” and prompt engineering as folklore. Now: context is the union of characterizations of every relevant entity in a user–application interaction, and context engineering is the function CE: (C, T) → f_context that optimizes how that context is collected, stored, managed, and used for a task T. This deliberately covers a 1995 sensor system and a 2025 agent with the same definition.
• The entropy-reduction thesis. Before: context tricks justified case-by-case. Now: one unifying principle — context engineering exists to bridge the entropy gap between high-entropy human intent and a machine’s limited assimilation capacity, and the more intelligent the machine, the less preprocessing you must do. This predicts why techniques rise and fall as models improve.
• A four-era evolutionary model (1.0 → 4.0). Before: “prompt era.” Now: a trajectory keyed to machine intelligence — 1.0 Primitive (structured inputs, rule triggers), 2.0 Agent-centric (natural language, ambiguity-tolerant; we are here), 3.0 Human-level (senses social/emotional cues), 4.0 Superhuman (machine constructs context for you). Useful as a planning compass: it tells you which problems are temporary (model will absorb them) vs. structural.
• A three-dimension design taxonomy with named patterns. Before: scattered blog posts. Now: a consolidated catalog — Collection & Storage, Management (textual/multimodal processing, layered memory, isolation, self-baking/abstraction), and Usage (intra/cross-system sharing, selection, proactive inference) — each with concrete patterns and the systems that use them (Claude Code, Letta, MemGPT, Manus, Gemini CLI). This is the genuinely reusable part for a builder.

How It Works (Technically)

Because this is a survey, the “mechanism” is the conceptual machine: the formal model of context plus the three design dimensions that operationalize it. I’ll demystify the math, then trace one real request through the whole pipeline.

The formal model, in plain English

The paper builds on Dey’s 2001 definition and makes it precise with four definitions. Don’t let the set notation scare you — it’s bookkeeping, not measure theory.

• Characterization Char: E → P(F). Read: every entity e (the user, the app, the terminal, a tool, the memory store, the model itself) maps to a set of facts that describe it. P(F) just means “a subset of all possible facts.” Operationally: Char(terminal) = {cwd, shell, recent commands}.
• Context C = ⋃_{e ∈ E_rel} Char(e). Read: context is the union of the fact-sets of all entities you decide are relevant (E_rel). The load-bearing word is relevant — choosing E_rel is a design decision, not a given. Pull in too little and the agent is blind; too much and you drown it.
• Context engineering CE: (C, T) → f_context, where f_context(C) = F(φ₁, φ₂, …, φₙ)(C). Read: given raw context C and a task T, produce a processing function f_context that is a composition F of operations φᵢ. The φᵢ are the verbs of the field: collect, store, format, fuse-multimodal, self-bake, select, share, adapt. F is how you wire them — sequential, parallel, conditional, iterative. That’s the entire paper in one equation: context engineering = choosing and composing the φ operators for your task.

There’s no loss function, no gradient, no training here — this is descriptive math that gives the field a shared coordinate system. The “what it computes” is: a pipeline that turns raw situational signals into the low-entropy token sequence your model actually conditions on.

The memory math (the one place with real operational content)

The layered-memory section adds three definitions worth translating, because they encode an actual policy you’d implement:

• Short-term memory M_s = f_short(c ∈ C : w_temporal(c) > θ_s). Plain English: keep the context items whose recency weight exceeds a threshold. w_temporal is “how recent/active is this?”; θ_s is your cutoff. This is your context window’s working set.
• Long-term memory M_l = f_long(c ∈ C : w_importance(c) > θ_l ∧ w_temporal(c) ≤ θ_s). Plain English: take the items that are old (fell out of the short-term cutoff) but important (importance weight above θ_l), and run them through f_long — which selects, abstracts, and compresses them into stable representations. The ∧ is just “and.”
• Transfer f_transfer: M_s → M_l. Plain English: the consolidation step — promote frequently-used or high-importance short-term items into long-term store. This is the agent equivalent of episodic→semantic memory in humans.

Operationally these three say: don’t keep everything; score each item by recency and importance, hold the recent stuff verbatim, and digest the important-but-old stuff into summaries. That digestion is what the paper calls self-baking — the difference between an agent that merely recalls and one that learns.

Architecture & data flow

flowchart LR
  subgraph Collect["1. Collect & Store"]
    S[Sensors / tools / dialogue<br/>multimodal signals] --> RAW[(Raw store<br/>SQLite / files / vectors)]
  end
  subgraph Manage["2. Manage"]
    RAW --> PROC[Textual + multimodal<br/>processing]
    PROC --> ORG[Organize: layered memory<br/>short-term vs long-term]
    ORG --> ABS[Abstract / self-bake:<br/>summaries, schemas, vectors]
  end
  subgraph Use["3. Use"]
    ABS --> SEL[Select relevant subset<br/>semantic + logical + recency]
    SEL --> WIN[[Context window]]
    WIN --> LLM((LLM / agent))
    LLM --> SHARE[Share across agents /<br/>systems]
    SHARE -.feedback.-> RAW
    LLM --> INFER[Proactively infer<br/>unspoken needs]
  end

Schematic of the paper's central thesis: as machine intelligence rises across the four eras, the entropy gap a human must manually close (the preprocessing "effort") shrinks. Drag the intelligence slider; watch how much raw human intent the machine can ingest directly vs. how much you must pre-digest.

A concrete trace

A user types "Search related documentation for me" into Gemini CLI.

1. Collect. E_rel = {user (the prompt), app (system instructions, GEMINI.md), environment (cwd), tools (search plugin), memory (session history), model service}. The CLI auto-loads static context (system prompt, ancestor/descendant GEMINI.md files) at startup and accumulates dynamic dialogue context during the session. Storage = the file system acting as a lightweight DB.
2. Manage. As the session grows, long histories are replaced with AI-generated summaries in a fixed format (goal, key knowledge, file-system state, recent actions, current plan) — that’s f_long / self-baking. The GEMINI.md hierarchy gives inheritance (home → project → subdir) and isolation.
3. Use. For this step, the agent selects the relevant subset — the search tool definition, the relevant project facts, the immediate dialogue — rather than dumping everything (the paper notes coding performance often degrades past ~50% window fullness). It calls the tool, integrates results, and could share that result with a sub-agent or write it back to memory.

Every arrow in the diagram is a φ operator you chose. The art is the composition.

The algorithm, simplified

There’s no single algorithm, so here’s the self-baking + select loop that is the operational heart of an Era-2.0 agent — the thing that lets it run for thousands of steps without overflowing its window.

# Era-2.0 context loop: hold recent verbatim, "bake" important-but-old into memory,
# and select a small relevant slice for each step. (llm/embed/search are stubs.)
def step(query, short_term, long_term, k=8):
    # 1. SELECT: assemble a small, relevant working set instead of dumping everything.
    #    score = semantic relevance + logical dependency + recency/frequency
    candidates = short_term + long_term
    ranked = sorted(candidates,
                    key=lambda c: 0.6*cos(embed(query), c.vec)   # semantic relevance
                                + 0.3*c.depends_on(query)        # logical dependency
                                + 0.1*c.recency,                 # recency/frequency
                    reverse=True)
    window = ranked[:k]                      # keep window well under ~50% full on purpose

    answer = llm(prompt=render(query, window))   # the actual model call
    short_term.append(Item(text=answer, vec=embed(answer), recency=now()))

    # 2. SELF-BAKE: when short-term grows, digest old+important items into long-term.
    #    This is the difference between recalling (storage) and learning (abstraction).
    if len(short_term) > SHORT_TERM_CAP:
        old = pop_oldest(short_term)
        if old.importance > THETA_L:                 # w_importance(c) > theta_l
            summary = llm(f"Distill the durable facts:\n{old.text}")  # f_long
            long_term.append(Item(text=summary, vec=embed(summary), recency=now()))
        # low-importance old items are simply dropped (Minimal Sufficiency Principle)

    return answer, short_term, long_term

The two principles the paper keeps returning to are baked in here: Minimal Sufficiency (collect/keep only what the task needs — value is in sufficiency, not volume) and Semantic Continuity (preserve meaning across compression, not raw bytes).

Built on Prior Work

The paper is explicitly a synthesis, and its honesty about lineage is a strength.

Results & Evidence

This is the honest part you need to hear up front: there are no experiments. Despite the extractor detecting an “experiments” heading, the paper presents no benchmarks, no ablations, no new system it built and measured. The evidence is (a) a conceptual argument for the framework and (b) a curated survey of what existing systems do. Treat every “result” as a claim about the field, not a measured number.

What it does establish well: a coherent vocabulary, a defensible historical lineage, and a thorough catalog of current practice with citations to real systems. The few quantitative claims it relays from others are useful rules of thumb but second-hand:

• AI coding quality often degrades past ~50% context-window fullness (cited from Osmani) — argues against “just use the big window.”
• Tool reliability for DeepSeek-v3 declines past ~30 tools, near-certain failure past ~100 (cited from Dbreunig) — argues for small, stable tool sets.
• Transformer attention is O(n²), the structural reason long context is expensive and reasoning thins out at scale.

What it does not establish: that its four-era model predicts anything testable, that its taxonomy is complete or better than alternatives, or that any recommended pattern beats another on a task. The 3.0/4.0 eras are admittedly speculative (“god’s eye view” AI). So: excellent map, zero head-to-head evidence. Use it to organize your thinking, not to settle a design argument with data.

How You’d Use It

For someone running an AI services company building agentic systems, the value is a diagnostic and sales framework, not a library to install.

• As a design checklist. Before building an agent, walk the three dimensions: What’s in E_rel? (collection), How do we keep the window lean over long runs? (management/self-baking), How do sub-agents share state and how do we select per-step? (usage). Most agent reliability bugs map to a missing answer here. This turns “the agent gets confused after a while” into a specific, addressable list.
• As a client-facing audit offering. “Context engineering audit” is a sellable artifact: score a client’s existing agent on collection / management / usage, point at the leaks (no memory layering, dumping full history each turn, no cross-agent protocol), and quote the fixes. The taxonomy gives you the rubric.
• For multi-agent orchestration (your ARC MAS experience maps directly). The cross-agent sharing patterns are the useful core: prompt-embedding (simplest, lossy), structured messages (schema-based, like Letta/MemOS), and shared memory/blackboard/graph (MemGPT, A-MEM, Task Memory Engine). Picking the right one is the difference between agents that cooperate and agents that talk past each other.
• For cost control. The KV-cache section is real money: keep prefixes byte-stable (no timestamps at the top of system prompts), make context updates append-only and deterministic, keep the tool list fixed and mask logits instead of swapping tools mid-run. These directly raise cache hit rate → lower latency and bill.

Build Your Own (Minimal Recipe)

The smallest thing that captures ~80% of the value is a context manager wrapping your LLM calls that implements layered memory + per-step selection. You don’t need the whole taxonomy.

Build order:

1. Raw store + working set. Append every turn (with an embedding) to a list. Cap the working set you send to the model — start by keeping the last N turns plus retrieved items, deliberately under ~50% of the window.
2. Selection. Implement the ranked select from the pseudocode: embed the query, score candidates by semantic similarity (a vector store / FAISS), add a recency boost, take top-k. This alone fixes most “agent ignores relevant info” complaints.
3. Self-baking. When the working set exceeds a cap, summarize the oldest important items with the model into a long-term note (fixed schema: goal / key facts / state / next steps), store the summary, drop the rest. This is the one part with real subtlety.
4. Cross-agent protocol (if multi-agent). Pick one sharing mechanism and standardize it — a JSON message schema is the pragmatic default.

The genuinely hard parts: (a) the importance/relevance scoring — recency is easy, “is this important” is not, and bad scoring quietly poisons long runs; (b) self-baking fidelity — summaries that drop the one fact you needed three steps later (Semantic Continuity is hard to verify). Libraries to reach for: a vector DB (FAISS/Chroma/pgvector), Letta/MemGPT or LangGraph for memory+orchestration scaffolding, and your model’s native prompt-caching to make stable prefixes pay off.

How to Improve It

Limitations are leverage. Concrete, testable directions:

1. Make the framework predictive, not just descriptive. The four-era model and the φ-taxonomy currently explain after the fact. Build a benchmark where you vary one φ (e.g., flat history vs. layered memory vs. self-baking) on the same long-horizon task and measure task success + tokens. That converts the paper’s taxonomy into design guidance with numbers.
2. Learned selection over hand-tuned weights. The select step uses fixed weights (0.6/0.3/0.1 in my sketch). Train a lightweight reranker (even a small cross-encoder, or an RL policy with task success as reward) to choose the context subset. This is exactly the “attention before attention” the paper hand-waves at — make it a learned policy.
3. Verifiable self-baking. Add a check that a summary preserves the facts needed downstream: after baking, run a QA probe over the original vs. the summary and reject summaries that lose answerable facts. Directly attacks the Semantic-Continuity failure mode.
4. Conflict-aware memory. The paper flags that large memories accumulate contradictions no one detects. Build a contradiction detector that flags when a new fact conflicts with a stored one, and a resolution policy (recency wins / source priority / ask user). This is a real product differentiator for long-lived agents.
5. Tie 3.0 ambitions to something measurable now. “Proactive need inference” is the most commercially interesting Era-3.0 idea. Operationalize it modestly: log when the agent correctly anticipated a follow-up the user then asked, and optimize for that hit-rate. Turns a speculative era into a metric.

Glossary

• Context engineering — systematically designing how context is collected, stored, managed, and used so a machine acts on human intent; formally CE: (C,T) → f_context.
• Entropy reduction — the paper’s core thesis: pre-digesting high-uncertainty human intent into a low-uncertainty form a machine can consume; the effort needed shrinks as machines get smarter.
• E_rel (relevant entities) — the subset of all entities (user, app, tools, memory, environment, model) whose facts you decide to include in context; choosing it is a core design act.
• φ operators — the verbs of context engineering: collect, store, format, fuse-multimodal, self-bake, select, share, adapt — composed by F into a pipeline.
• Self-baking — periodically digesting raw context into compact persistent structures (summaries, schemas, vectors); the line between an agent that recalls and one that learns.
• Layered memory — separating short-term (recent, verbatim, in-window) from long-term (old-but-important, abstracted) memory, with a transfer/consolidation step between them.
• Context isolation / sub-agent — giving a specialized agent its own context window, prompt, and tool permissions so it can’t pollute the main conversation.
• Minimal Sufficiency Principle — collect/keep only what the task needs; value is in sufficiency, not volume.
• Semantic Continuity Principle — compression must preserve meaning across steps, not just retain raw data.
• KV cache — stored attention keys/values for past tokens so they aren’t recomputed; stable, append-only prefixes raise the hit rate and cut latency/cost.
• RAG (retrieval-augmented generation) — fetching relevant documents/chunks (usually via vector similarity) and adding them to the prompt before generation.
• Blackboard / shared memory — a common store where agents read/write to coordinate indirectly, instead of passing messages point-to-point.
• Era 1.0–4.0 — the paper’s intelligence-keyed stages: Primitive (rules/sensors), Agent-centric (today’s LLMs), Human-level (senses social/emotional cues), Superhuman (machine constructs context for you).
• O(n²) attention — the quadratic cost of standard transformer attention in sequence length; the structural reason long contexts are slow and reasoning thins out at scale.