Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models

TL;DR

LLM agents and domain experts increasingly improve themselves by changing their context (prompts, memory, evidence) rather than their weights. The problem: existing methods either compress everything into short generic prompts (losing hard-won detail) or rewrite the whole context each step with an LLM, which catastrophically “collapses” accumulated knowledge into a useless summary. ACE fixes both by treating context as a playbook of itemized bullets with three specialized roles — a Generator that solves tasks, a Reflector that extracts lessons, and a Curator that merges small delta updates into the playbook using cheap non-LLM logic. The result is +10.6% on agent tasks (AppWorld), +8.6% on financial reasoning, an 86.9% drop in adaptation latency, and parity with the top proprietary leaderboard agent — all learned from execution feedback alone, no labels required.

Problem & Motivation

The pain in one sentence: when you let an LLM keep rewriting its own memory/prompt, it keeps making it shorter and dumber until performance crashes.

Context adaptation — improving an agent by editing its inputs instead of retraining — is attractive because it’s interpretable, instant, and shareable across models. But the authors document two concrete failure modes in current approaches:

1. Brevity bias. Prompt optimizers (GEPA, MIPROv2) converge toward short, generic instructions because brief prompts score well on validation and are easy to search over. GEPA even markets brevity as a feature. But for a tool-using agent, “be helpful and write good code” throws away the exact API quirks, error patterns, and tool-call recipes that actually move the needle. Gao et al. saw optimizers repeatedly emit near-identical bland instructions like “Create unit tests to ensure methods behave as expected.”

2. Context collapse. When you ask an LLM to fully rewrite a large accumulated context each step, it tends to compress. The paper’s killer example on AppWorld: at step 60 the context was 18,282 tokens and scored 66.7% accuracy; at step 61 the LLM rewrote it down to 122 tokens and accuracy fell to 57.1% — worse than having no adaptive context at all (63.7%). One bad rewrite erased everything.

Their thesis flips the usual instinct: humans like concise generalizations, but LLMs do better with long, detailed, messy context and can pick out what’s relevant at inference time. So don’t compress — accumulate, and let the model decide what matters.

What’s New (Core Contribution)

• Context-as-playbook, not context-as-prompt. Before: context is one monolithic string an LLM rewrites wholesale. Now: context is a collection of structured, individually-addressable bullets (each with an ID, helpful/harmful counters, and a small unit of content). This is what makes localized edits and de-duplication possible.
• Three-role division of labor (Generator / Reflector / Curator). Before: Dynamic Cheatsheet used one model to both solve and update memory. Now: the Reflector is a dedicated step that separates evaluating what went right/wrong from deciding how to edit the context. Ablations show this separation alone is a major contributor.
• Incremental delta updates with non-LLM merging. Before: every update is a full LLM rewrite (slow, expensive, collapse-prone). Now: the Reflector proposes small “delta” bullets; a deterministic, lightweight merge function appends/updates them. Deltas can be batched and merged in parallel. This is the source of the 80–90% latency/cost reductions.
• Grow-and-refine. Before: nothing stops the playbook from bloating or duplicating. Now: new bullets append, existing ones update in place (counters), and a semantic-embedding de-duplication step prunes redundancy — run eagerly or lazily when the window fills.

The honest read: the individual ingredients aren’t all new (Dynamic Cheatsheet pioneered adaptive memory; Reflexion pioneered self-reflection). The genuine contribution is the engineering synthesis — itemized structure + role separation + deterministic delta merging — that makes self-improving context actually scale without collapsing.

How It Works (Technically)

ACE runs an adaptation loop. There’s no gradient, no reward function, no weight update anywhere — this is entirely a prompting/memory architecture. The “learning” is the playbook getting better.

The three roles:

• Generator — the actual agent. Given a query and the current playbook, it produces a reasoning trajectory (ReAct-style: think, call tools, observe, repeat). Crucially it also flags which bullets it used and whether they helped or misled.
• Reflector — looks at the trajectory plus any feedback signal (did the code run? did the answer match? did the environment error?) and distills concrete lessons: “API X needs auth header Y”, “this task type fails when you skip step Z”. It can iterate several rounds (default max 5) to refine its lessons.
• Curator — converts the Reflector’s lessons into delta items (small candidate bullets, each tagged add/update/which-section) and merges them into the playbook using deterministic, non-LLM logic — string IDs, counter increments, embedding-based de-dup. No big rewrite, so nothing can collapse.

The feedback signal is the secret sauce. ACE doesn’t need ground-truth labels because in agent settings the environment gives feedback for free: code either executes or throws, an API call succeeds or 404s. The Reflector turns those raw signals into language lessons. (The flip side, shown in results: when there’s no reliable signal — neither labels nor execution outcomes — ACE can pollute its own context with spurious lessons and degrade.)

Why no collapse: collapse happens because “rewrite this whole 18k-token context” gives the LLM license to summarize. ACE never asks for a rewrite. The LLM only ever emits small deltas; the existing playbook is preserved by deterministic code, not by the LLM’s discretion. Information can only be removed by the explicit de-dup/prune step, not accidentally lost in a summary.

A bullet, concretely:

[id: ap-0412 | helpful: 7 | harmful: 0 | section: tool_use]
When calling the email API, list_messages requires `folder` param;
omitting it returns an empty list, not an error — check folder first.

Counters let grow-and-refine keep what’s earning its place and prune what isn’t.

Architecture & data flow

flowchart LR
  Q[New query] --> G[Generator: solve task ReAct-style]
  PB[(Playbook: itemized bullets)] --> G
  G -->|trajectory + bullet usage flags| R[Reflector: distill lessons]
  FB[Execution feedback / labels] --> R
  R -->|iterate up to 5x| R
  R -->|lessons| C[Curator: form delta items]
  C -->|deterministic merge: append / update counters / dedup| PB
  G --> OUT[Answer]
  PB -.grow-and-refine: embed + prune redundancy.-> PB

Schematic of context collapse vs. ACE's incremental growth, built from the paper's AppWorld numbers (step 60: 18,282 tokens / 66.7%; step 61 collapse: 122 tokens / 57.1%; no-context baseline 63.7%). Toggle the methods to see why monolithic rewrite is fragile and delta-append is not.

The algorithm, simplified

# ACE adaptation loop. No gradients, no rewards — the "model" that improves is `playbook`.
# Stubs: llm(prompt)->str, embed(text)->vec, run_agent(query, ctx)->Trajectory
def ace_step(query, playbook, get_feedback):
    # 1. GENERATOR: solve using current playbook; it also flags which bullets it leaned on
    traj = run_agent(query, ctx=render(playbook))      # ReAct trajectory + bullet usage flags

    # 2. feedback is FREE in agent settings: did the code run? did the answer match?
    signal = get_feedback(traj)                         # execution outcome OR ground-truth label

    # 3. REFLECTOR: turn trajectory + signal into concrete lessons (can iterate to sharpen)
    lessons = traj.usage_flags
    for _ in range(5):                                  # max refinement rounds
        lessons = llm(f"Critique this run and extract reusable lessons.\n"
                      f"trajectory={traj}\nsignal={signal}\nprev={lessons}")
        if lessons.converged: break

    # 4. CURATOR: emit SMALL deltas (add/update bullets) — never a full rewrite
    deltas = llm(f"Convert lessons into itemized bullet edits.\nlessons={lessons}")

    # 5. MERGE deterministically (NON-LLM) — this is what prevents context collapse
    for d in deltas:
        if d.id in playbook:
            playbook[d.id].helpful += d.helpful; playbook[d.id].harmful += d.harmful
        else:
            playbook[d.id] = d                          # append a new bullet
    return playbook

def grow_and_refine(playbook, max_tokens):
    # prune redundancy by semantic similarity; run eagerly or lazily when window overflows
    vecs = {b.id: embed(b.content) for b in playbook.values()}
    drop = find_near_duplicates(vecs, keep=lambda b: b.helpful - b.harmful)  # keep the proven one
    for bid in drop: del playbook[bid]
    return playbook

Offline vs. online: offline runs this over a training split (optionally multiple epochs — revisit the same queries to strengthen the playbook) to produce a system prompt. Online runs it sequentially at test time: predict with the current playbook, then update from that sample. Offline warmup (build a playbook offline first, then keep adapting online) gave the best agent numbers.

Built on Prior Work

Results & Evidence

Agents (AppWorld, base model DeepSeek-V3.1):

• Offline: ReAct + ACE hits 59.4% average vs. 46.0% (ICL) and 46.4% (GEPA) — +12.3% / +11.9%.
• Online: ACE 66.0% avg vs. Dynamic Cheatsheet 52.3% — +13.7% (and +7.6% in the headline aggregate framing).
• No labels: ACE still gets +14.8% over the ReAct baseline using only execution feedback.
• Leaderboard: ReAct + ACE (59.4%) ≈ top-ranked IBM CUGA (60.3%, GPT-4.1-based) on average, and beats CUGA on the harder test-challenge split (+8.4% TGC) — using a smaller open model.

Finance (FiNER + Formula, XBRL reasoning):

• Offline with labels: +10.9% avg over ICL/MIPROv2/GEPA; ACE 76.6% avg.
• Online with labels: +6.2% over Dynamic Cheatsheet.

Cost/speed (the strongest practical case):

• Offline AppWorld vs. GEPA: −82.3% latency, −75.1% rollouts.
• Online FiNER vs. DC: −91.5% latency, −83.6% token dollar cost.

Ablations: removing the Reflector + multi-epoch drops offline AppWorld from 59.4% to 55.1%; offline warmup lifts online from 56.1% to 59.5%. Each piece earns its keep.

Caveats / what it does NOT establish:

• Feedback-dependent. Table 2 shows that without labels or execution signals, both ACE and DC can degrade below baseline (e.g., online ACE w/o labels: 72.9% with some splits dropping). Context can be polluted by bad lessons. ACE is not a free lunch in label-poor, signal-poor domains.
• Narrow domains. Two task families (agents, financial XBRL). No coding-at-large, no creative, no multi-agent-coordination eval.
• Same model for all three roles (deliberately, for fairness) — so the paper doesn’t show the likely-better setup of a strong Reflector + cheap Generator.
• “Longer context isn’t more expensive” rests on KV-cache reuse infrastructure you may or may not have in production.
• Most numbers come from one base model (DeepSeek-V3.1); generality across model families is asserted, not broadly shown.

How You’d Use It

For an AI services company, ACE is a concrete, sellable capability: agents that get measurably better at a client’s specific environment over time, with an auditable, human-editable memory. Where it slots in:

• Replace your “prompt tuning” engagements. Instead of hand-crafting system prompts per client, stand up an ACE loop that mines the client’s own execution traces into a playbook. The interpretability is a selling point: you can show the client the exact bullets the agent learned (“it discovered your ERP API needs X”).
• Self-improving support / ops agents. Any agent operating in an environment with natural feedback (CI/CD pass-fail, API success, ticket resolution) can accumulate domain-specific tactics without you fine-tuning anything.
• Compliance-friendly memory. Because the playbook is human-readable bullets with IDs, you can do selective unlearning — delete a bullet for GDPR/CCPA or when a domain expert flags it wrong. Try doing that with fine-tuned weights.
• In a multi-agent system (your ARC MAS experience): the playbook is a shared, append-only knowledge artifact. Multiple Generator agents can read it; their deltas merge in parallel deterministically — which is exactly the kind of conflict-free coordination that’s painful with free-text shared memory.

Effort to stand up a useful v1: low-to-moderate. The hard parts are (a) wiring a reliable feedback signal and (b) the de-dup step. Everything else is prompt plumbing.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

1. Playbook store. A list of bullets: {id, text, helpful, harmful, section}. JSON file or a vector DB. Start with a dict.
2. Generator. Your existing ReAct/tool-calling agent, but inject the rendered playbook into the system prompt and ask it (in the same call or a cheap follow-up) to name which bullets it used and whether they helped.
3. Feedback hook. The single most important piece. Wire the cheapest reliable signal you have: exit codes, test results, API status, or labels if you have them.
4. Reflector. One LLM call: “Here’s the trajectory and the outcome. What 1–5 reusable lessons should we remember?” Skip the 5-round refinement for v1.
5. Curator + deterministic merge. One LLM call to turn lessons into add/update bullet ops, then plain Python to apply them. Do NOT let the LLM rewrite the whole playbook — that’s the entire point.
6. Grow-and-refine. Embed bullets (any embedding model), cosine-dedup above a threshold, keep the one with the higher helpful - harmful. Run lazily when you exceed a token budget.

Reach for: any tool-calling LLM (DeepSeek-V3.1, GPT-4-class, Claude), an embedding model for de-dup, and a long-context-friendly serving setup with prompt/KV caching so the growing playbook stays cheap to prefill.

The two genuinely hard parts: (1) designing a feedback signal that’s reliable enough not to poison the playbook, and (2) de-dup tuning — too aggressive and you re-introduce brevity bias; too loose and the playbook bloats.

How to Improve It

1. Asymmetric roles. The paper deliberately used one model for all three roles. Use a strong reasoner (o3/Claude) as Reflector and a cheap fast model as Generator — likely better quality at lower cost. Easy to test.
2. Confidence-weighted / decaying counters. Helpful/harmful counters are unweighted and monotonic. Add recency decay and weight by task difficulty so stale or easy-task lessons don’t dominate. Testable via ablation on the existing benchmarks.
3. Verifier before merge (anti-pollution). The biggest weakness is label-free degradation. Add a lightweight verifier or self-consistency check that gates a delta before it’s merged — only commit a lesson if it reproduces. Directly attacks the Table-2 failure.
4. Retrieval over the playbook. As playbooks grow huge, inject only the top-k relevant bullets per query (the paper notes “fine-grained retrieval” as a property but feeds the whole thing). Pairs naturally with the embeddings you already compute for de-dup.
5. Cross-client / cross-task playbook transfer. Test whether a playbook learned in one environment seeds faster learning in a related one — a real product moat (your agents arrive pre-skilled). Untested in the paper.
6. Conflict resolution for parallel deltas. Deterministic merge handles appends, but contradictory lessons (bullet A says “do X”, bullet B says “never X”) aren’t reconciled. Add a contradiction-detection pass during grow-and-refine.

Glossary

• Context adaptation / context engineering — improving an LLM by editing its inputs (prompt, memory, evidence) instead of its weights.
• Brevity bias — the tendency of prompt optimizers to converge on short, generic prompts that score well but drop domain detail.
• Context collapse — when an LLM asked to rewrite a large context compresses it into a short, information-poor summary, crashing performance.
• Playbook — ACE’s term for the context: a growing, structured collection of itemized strategy bullets.
• Bullet / context item — one memory unit: metadata (ID, helpful/harmful counters) + content (a strategy, concept, or failure mode).
• Generator / Reflector / Curator — the three roles: solve the task / extract lessons / merge lessons into the playbook.
• Delta update — a small set of candidate bullet edits, merged in by deterministic code rather than a full LLM rewrite.
• Grow-and-refine — append new bullets, update existing in place, then prune duplicates via embedding similarity.
• Offline vs. online adaptation — build the context on a training split first (offline, → system prompt) vs. update it sample-by-sample at test time (online, → live memory).
• Multi-epoch / offline warmup — revisit training queries multiple passes; or build offline then continue adapting online.
• ReAct — agent loop interleaving reasoning (“thoughts”) with tool/action calls and observations.
• Execution feedback — free signal from the environment (code ran/failed, API succeeded) used instead of ground-truth labels.
• TGC / SGC — Task Goal Completion / Scenario Goal Completion, AppWorld’s success metrics.
• XBRL — eXtensible Business Reporting Language, the structured format for financial filings used in the FiNER/Formula benchmarks.
• KV-cache reuse — serving optimization that caches a context’s computed key/value tensors so re-using a long prompt doesn’t re-pay the prefill cost.
• GEPA / MIPROv2 / Dynamic Cheatsheet — baseline context/prompt-optimization methods ACE compares against.