TL;DR
Most harness comparisons pit whole systems against each other (OpenHands vs SWE-Agent vs Claude Code), so nobody can tell which part made the difference. The authors build one small, modular ReAct harness and turn three knobs one at a time: context management (five strategies, four window sizes), planning (on/off), and action space (eight typed tools vs bash only). They run 176 matched settings across four open models (Nemotron-3 30B/120B/550B and Mistral-Medium-3.5-128B) on SWE-Bench Verified and Terminal-Bench 2.1. The headline is that every component is conditional: context management is worth +35.7 points of success at a 32k window and only +2.7 at 128k; a cheap “elide first, summarize only if still too big” policy gives the best cost at equal accuracy; planning is an accuracy crutch for the weakest model but a cost cutter (-30%) for strong ones; and a bash-capable model gets higher success at half the cost with bash alone. Harness design is a per-model, per-budget, per-task choice, not a list of defaults.
Problem & Motivation
You pick a harness, swap in a new model, and the score moves. Was it the tool set? The context policy? The planner? With whole-system comparisons you cannot know. Cao et al. (2026) found Claude-Opus-4.5 does best in OpenHands while Claude-Sonnet-4.5 does best in SWE-Agent. That result says “harness preference depends on the model” but gives you nothing to act on, because each harness bundles a dozen choices.
The practical pain for a builder is concrete:
- You add summarization, recall, sub-agents, a planner, and twenty tools because each one “helped in some paper”, and your cost per task triples.
- You upgrade the model and keep the scaffolding that the old model needed, paying for crutches the new one does not use.
- You run a small model on a cheap 32k window and cannot tell whether failures come from the model or from the history simply overflowing.
Prior ablation work either studied short-horizon reasoning with prompt-level scaffolds (Liu 2026, no context pressure), or reported architecture preferences without an explicit context-window sweep. Nobody had varied planning, the action interface, and context management together on long coding trajectories, across both window size and model size.
What’s New (Core Contribution)
This is an empirical paper. The harness itself is deliberately ordinary. The contribution is the controlled measurement.
- A component-isolating harness. Before: harnesses couple many choices so ablations are local and messy. Now: a fixed ReAct loop (LangGraph, run through Harbor containers) where planning, action space, and context management are independent switches, and safety gates, post-edit lint, and stuck detection are held constant. This is genuinely useful as an experimental method, even if no single piece is new.
- A tiered decomposition of context management. Before: papers introduce one compaction method and test it on one model. Now: three primitive mechanisms (M1 elision, M2 recall, M3 summarization) combined into five tiers T0 to T4, so the value of each mechanism is a matched comparison. T4 (elide at a soft threshold, summarize only at a hard threshold) is the staged policy they recommend.
- A window-size sweep that explains why context management helps. Before: “summarization improves performance.” Now: almost all of the benefit is preventing context-overflow termination, and it shrinks from +35.7 to +2.7 points as the window grows from 32k to 128k.
- Capability crossovers for planning and tools, explained by trajectory analysis. Before: “planning helps”, “good tools help.” Now: planning and predefined tools both flip from accuracy aids to overhead as the model gets stronger, and an LLM judge (94% agreement with humans, kappa 0.93) labels every turn by phase to show the behavioral reason.
What is not new: elision, MemGPT-style recall, rolling summarization, update_plan tools, and the ReAct loop are all borrowed. Do not read this as a new method paper.
How It Works (Technically)

There is no new algorithm to learn here, so this section explains the experimental machine: what the harness does each turn, what each knob changes, and how the results are compared.
The loop. Each turn: assemble input (system prompt + task + action schemas + current plan + managed history), call the model, execute tool calls in a Docker/E2B container, append (reasoning, action, observation) to history H, then run context management on H. Up to 300 steps per task, 16k output tokens per turn, temperature 0. Tool results are cut at 24k characters, and up to eight read-only tools can run in parallel in one step.
Knob 1: planning. When on, a system instruction defines a plan protocol, the first turn asks for a plan before acting, and the model edits it with an update_plan tool. The plan is re-injected into every turn’s input but not stored in the history, so compaction never deletes it. When off, all of that is removed. This tests a persistent plan scaffold, not “planning” as a reasoning style.
Knob 2: action space. Full set = read_file, write_file, edit_file, list_files, glob_files, grep_text, web_fetch, bash. Bash-only removes everything except bash (plus update_plan and recall_event when their components are on). Note the bundle: the file tools also carry read-before-write checks, file-state tracking, and automatic lint after edits. So “tools vs bash” compares two whole interfaces, not just tool count.
Knob 3: context management. History is split into three regions: the preamble (system prompt + task, never touched), a recent window (30% of the usable window, at least the last two turns, never touched), and the middle (the only region that gets compacted). Three mechanisms act on the middle:
- M1 elision replaces the body of a bulky old tool observation with a short stub. Cheap, rule-based, lossy.
- M2 recall saves the original body to disk before eliding it and exposes
recall_event(id)so the model can pull it back. Lossless, but only if the model asks. - M3 summarization makes a separate, tool-free call to the same model to fold the oldest middle events into a running summary. Expensive, lossy, but compresses a lot.
Two thresholds drive T4: soft B1 = 0.6 and hard B2 = 0.85 of the usable window. In plain English: “Once history is 60% full, throw away the bulky old tool outputs (keeping copies on disk). If it is still 85% full after that, pay for a summary of the oldest stuff.” T1 to T3 each have one mechanism, so they fire only at B2.
| Tier | M1 elide | M2 recall | M3 summarize | When it acts |
|---|---|---|---|---|
| T0 | no | no | no | never; run dies on overflow |
| T1 | yes | no | no | at B2 |
| T2 | yes | yes | no | at B2 |
| T3 | no | no | yes | at B2 |
| T4 | yes | yes | yes | elide at B1, summarize at B2 |
Held fixed. A workspace guard rejects any path that escapes the project root (including via symlinks); read-before-write with content hashes; an allow/ask/deny permission layer; tool errors returned as observations, never raised; ruff/pyflakes lint appended after every Python edit; stuck detection that nudges after five identical calls and kills the run after eight identical failing calls.
The comparison math, demystified. Each setting runs once per task. To decide if, say, T4 beats T0, they use an exact McNemar test: look only at the tasks where the two settings disagree (one solved, one failed) and ask whether the split is more lopsided than a coin flip. That is the right test for paired pass/fail outcomes on the same tasks. Because they run many comparisons, they apply Benjamini-Hochberg correction: sort the p-values and accept only those below a sliding bar, so that at most about 5% of the “significant” findings are expected to be false alarms. An asterisk in their tables means it survived that bar.
One trace. Nemotron-3 30B gets a SWE-Bench issue with a 32k window. Under T0, it reads three source files and a test log; the window fills by around turn 20-30 while it is still localizing the bug, and the run terminates with an overflow error. No edit is ever made. Under T4, at 60% full the old file dumps become stubs and copies go to disk; the model keeps localizing, edits a file, gets a lint warning appended, fixes it, runs tests. At 85% the oldest middle turns become a summary. The run lasts far longer (median 50 to 180 turns depending on model and tier) and reaches the Verify phase. Success goes from 9.4% to 21.2%. The model’s behavior (phase order and mix) is about the same; it just gets to keep going.
Architecture & data flow
flowchart LR
subgraph Input["Input assembly (per turn)"]
SP[System prompt + task]
AS[Action schemas<br/>full set or bash-only]
PL[Current plan<br/>injected, not stored]
MH[Managed history H]
end
SP --> LLM[Coding model]
AS --> LLM
PL --> LLM
MH --> LLM
LLM -->|tool calls| G[Safety gates<br/>root guard, read-before-write, permissions]
G --> ENV[Container: files, shell, web]
ENV -->|observation + lint| ST[Stuck detector]
ST -->|append turn| H[(History H)]
H --> CM[Context manager T0-T4]
CM --> MH
CM <-->|offload / recall_event| DISK[(External store)]
flowchart TD
A[Append think, action, observation to H] --> B{Model called recall_event this turn?}
B -->|yes| C[Read original back from store]
B -->|no| D[Split H: preamble, middle, recent window]
C --> D
D --> E{tokens H >= B1 = 0.6?}
E -->|no| Z[Send H to model]
E -->|yes| F[M1+M2: stub bulky middle outputs, save originals]
F --> G{tokens H >= B2 = 0.85?}
G -->|no| Z
G -->|yes| S[M3: summarize oldest middle events]
S --> Z
Schematic simulator of a single trajectory filling its context window. Pick a tier and a window size, then press Run. Each turn adds a tool observation. Watch where T0 dies, where elision (amber) and summarization (purple) fire, and how many costly summary calls each tier makes. Numbers are illustrative, not the paper's data.
The algorithm, simplified
SOFT, HARD, RECENT = 0.60, 0.85, 0.30 # fractions of the usable window
def manage_context(H, window, store, tier="T4"):
"""Called after every turn. H = list of events (role, kind, text, id)."""
pre, mid, recent = split(H, keep_recent=max(2, budget_turns(H, RECENT * window)))
used = tokens(H)
elide_at = SOFT if tier == "T4" else HARD # single-mechanism tiers wait for the hard line
if tier in ("T1", "T2", "T4") and used >= elide_at * window:
for ev in mid:
if ev.kind == "tool_obs" and tokens(ev.text) > 500:
if tier in ("T2", "T4"):
store[ev.id] = ev.text # M2: keep a copy, model may recall_event(id)
ev.text = f"[elided output {ev.id}; call recall_event to restore]" # M1
used = tokens(pre + mid + recent)
if tier in ("T3", "T4") and used >= HARD * window:
oldest, mid = mid[:len(mid) // 2], mid[len(mid) // 2:]
summary = llm(SUMMARY_PROMPT + render(oldest), tools=None) # M3: paid call, same model
mid = [event("summary", summary)] + mid
if tier == "T0" and used > window:
raise ContextOverflow # T0: the run simply ends here
return pre + mid + recent
The whole point of T4 is the ordering: the free, rule-based step runs first and often brings the history back under the hard line, so the expensive summary call happens less. That ordering, not any one mechanism, is what buys the lower cost.
Built on Prior Work
| Prior idea | What it gave | What this paper changes |
|---|---|---|
| ReAct (Yao et al. 2022) | think / act / observe loop | Used as the fixed substrate; not varied |
| SWE-Agent (Yang et al. 2024) | Agent-computer interfaces; “the harness, not the model alone, governs performance” | Splits the interface question out and shows it flips with model skill |
| OpenHands, Claude Code, Codex | Production harnesses with bundled designs | Studied indirectly: shows why cross-harness rankings conflate mechanisms |
| Agentless, AutoCodeRover | Simple localize / repair / validate pipelines can match elaborate agents | Consistent finding: strong models need less scaffolding |
| MemGPT (Packer et al. 2023), LCM | Lossless memory via external store + retrieval | Tests it as M2; models almost never use it |
| Prompt compression, StreamingLLM, recursive summarization | Lossy elision and summaries | Tests as M1 and M3; shows staging them is the efficient combination |
| AgentArch, Mehtiyev and Assuncao 2026 | Model-specific architecture preferences | Adds a context-window sweep and within-family scale axis |
| Liu 2026 full-factorial scaffolding study | Factorial design on short reasoning tasks | Moves to long-horizon coding where the window actually binds |
Results & Evidence


This is the heart of an empirical paper, so here are the numbers and how much weight they can bear.
1. Context management is an overflow fix, and its value scales inversely with the window. Managed tiers minus T0, averaged across models:
| Window | SWE-Bench gap (pts) | Terminal-Bench gap (pts) |
|---|---|---|
| 32k | +35.7 | +9.5 |
| 64k | +15.9 | +7.5 |
| 96k | +5.5 | +4.8 |
| 128k | +2.7 | +2.8 |
T0’s overflow rate falls from 78.7% to 8.7% on SWE-Bench (61.0% to 12.1% on Terminal-Bench) over the same sweep, and managed tiers overflow on exactly zero tasks. The gap tracks the overflow rate almost perfectly. The most striking cell: Nemotron-3 550B at 32k goes from 6.4% (T0) to 55.6% (T4). A strong model with no context policy on a small window is nearly useless.
Success rate vs context-window budget, from the paper's Tables 3 and 4. Gray = T0 (no management), blue = T4. Toggle model and benchmark. The shaded area is the value of context management; watch it close as the window grows.
2. T4 wins on cost, not accuracy. Success rates across T1 to T4 are close at every window. T4 has the lowest cost in seven of eight model-benchmark panels and the lowest average peak-context ratio at every window, because early elision handles most of the pressure before summaries are needed. Example: 550B on SWE-Bench at 32k, T1 costs $2.57/task and T4 costs $1.45 for a slightly higher success rate. T3 (summarize only) is sometimes the top scorer (550B at 32k: 58.4%) but costs more summary calls.
3. Recall (M2) is dead weight. T2 vs T1 (identical except recall): T2 wins 15, loses 14, ties 3 of 32 comparisons; mean difference -0.36 points. 56% of T2/T4 settings never call recall_event; at 128k the mean is 0.007 calls per task. Almost all usage is the 30B model under max pressure, and even its heaviest-use setting (4.3 calls/task) scored 3.4 points below elision alone. If you have built a “lossless” memory store into your harness, this is evidence the model is not using it.
4. Planning flips with capability (T4/128k).
| Model | SWE: SR change, cost change | Terminal-Bench: SR change, cost change |
|---|---|---|
| Nemotron-3 30B | +11.6 pts, cost up ~4x | +4.5 pts, cost up |
| Nemotron-3 120B | -2.6 pts, cost +36% | 0 pts, cost -26% |
| Nemotron-3 550B | -2.0 pts, cost -30% | -1.1 pts, cost -4% |
| Mistral-Medium-3.5 | -0.4 pts, cost -32% | -2.2 pts, cost -40% |
The mechanism (from turn-level labels): without planning, 30B’s median SWE trajectory collapses from 40 turns to 5; 68.6% of runs end without any edit and 58.4% die still in Localize (vs 27.8% and 10.4% with planning). For 550B, planning cuts median length from 108 to 74 turns and the removed turns are almost all post-edit verification. The plan gives the strong model a “done” signal.
5. Tools vs bash flips with capability and task type (T4/128k, planning on).
| Model | SWE: full tools vs bash | Terminal-Bench: full tools vs bash |
|---|---|---|
| Nemotron-3 30B | tools +15.0 pts | tools +10.1 pts |
| Nemotron-3 120B | tools +1.6 pts | tools +4.5 pts |
| Nemotron-3 550B | bash +3.6 pts, cost -53% | bash +5.6 pts, cost -30% |
| Mistral-Medium-3.5 | tools +23.2 pts | bash +6.7 pts |
Why the weak model fails with bash only: it keeps emitting calls to read_file and friends it learned in training; the harness cannot resolve them, and 66% of its Terminal-Bench bash-only runs end that way (average trajectory falls from 71 turns to 15). Why the strong model wins: it writes denser, composite shell commands, issues 24-32% fewer calls, and re-patches already-edited files far less (550B: 4.6 to 1.5 re-patches per task). Create-or-replace share of file writes jumps from 51% to 76%. Mistral shows the task boundary: on shell-heavy Terminal-Bench it already sends 72% of actions through bash, so removing the other tools helps; on SWE-Bench, 32.8% of its bash-only runs never edit a file at all.
The capability crossover, plotted from the paper's T4/128k numbers. Each dot is a model: x = cost change, y = success change, when you turn a component ON (planning on, or full tool set instead of bash). Top-left = helps and saves money; bottom-right = hurts and costs more. Toggle the component and benchmark.
What the evidence does NOT establish:
- One run per task. Terminal-Bench has 89 tasks, so a 4-point swing is three or four tasks. Many Terminal-Bench cells are not significant; the authors lean on consistent direction across models. Treat Terminal-Bench numbers as directional.
- Planning and tool ablations only at T4/128k. No factorial design. Planning’s effect at 32k is unknown, and so are interactions (does planning help more when summaries are erasing context?).
- Open models only, served locally. No frontier closed models. “Bash-capable” here means Nemotron-550B; whether the crossover holds for models heavily trained on a specific tool schema (which most frontier coding models are) is untested.
- Model size is a loose proxy for skill. Mistral-128B behaves “strong” on SWE and “bash-native” on Terminal-Bench, so the real variable is training exposure to interfaces, not parameter count.
- One implementation of each component. One plan prompt, one threshold pair (0.6/0.85), one summarizer prompt. A better recall prompt or a plan protocol tuned for strong models might change the verdict.
- SWE-Bench Verified is Python-only, and the post-edit lint only covers Python.
- Cost is token-priced at OpenRouter rates, not wall-clock. The 30B model’s planning cost “increase” is from $0.02 to $0.09 per task; the percentages look dramatic on tiny numbers.
How You’d Use It
This paper’s value is as a decision guide for harness knobs you already have. Four lenses fit.
Your harness. Treat every component as a per-model setting, not a feature.
- Where it slots in: your agent config. Make
context_policy,planning, andtool_setmodel-profile fields instead of global constants. - What it improves: you stop paying for scaffolding your current model does not need. The 550B result (bash only: higher success, half the cost) is the kind of saving you only get by switching a knob off.
- What it costs: a small matched eval (the same 50-100 tasks, each config once) every time you change models. That is a day of compute, not a research project.
- What must be true: you can run your own tasks in a sandbox with a pass/fail check. If you cannot measure, you cannot use any of this.
Your context policy specifically. Adopt the T4 shape, drop the recall store unless you prove you need it.
- Where it slots in: the function that runs between turns in your loop. Protect system prompt + task + last N turns; elide bulky old tool outputs first at ~60% full; summarize only if still above ~85%.
- What it improves: cost and peak context, at equal accuracy. On small windows or small local models, it is the difference between working and not working.
- What it costs: a token counter and about 60 lines of code. The summarizer call is the only paid part.
- What must be true: tool outputs are the bulk of your history (true for coding and most tool-heavy agents). If your history is mostly model reasoning, elision has less to cut.
Your applications. If you ship an agent on a small or local model with a short window (on-device, a cheap 32k endpoint, a fine-tuned 8B), context management is not optional; it is the main thing standing between you and 9% success. If you ship on a frontier model with 200k+ context, expect context management to matter mostly for cost and for the long-tail tasks that blow past the window.
Your workflows and methodologies. Change how you evaluate harness changes.
- Compare configs on the same tasks with a paired test (McNemar), not by eyeballing two success rates.
- Log per-turn phase (localize / reproduce / fix / verify) with a cheap LLM judge. The paper’s most useful insights came from where trajectories stopped, not from the scores. “Planning kills post-edit verification loops” is the kind of thing you only see in phase plots.
- When you upgrade a model, re-run your ablations. The study’s core message is that the right harness moves with the model.
Build Your Own (Minimal Recipe)
Goal: a harness where you can flip the same three knobs and measure on your own tasks.
- Fixed loop (half a day). A plain ReAct loop over the model’s native tool-calling API. Return tool errors as observations. Cap steps. Truncate tool outputs to a fixed character budget.
- Two tool registries (half a day).
FULL = [read_file, write_file, edit_file, list_files, glob_files, grep_text, bash]andBASH = [bash]. Give the file tools read-before-write checks and run a linter after edits. Important trap from the paper: in bash-only mode, models will still emit calls to tools that are not registered. Decide on purpose whether you return “unknown tool, use bash” as an observation (kinder, and probably what you want) or end the run (what effectively killed the paper’s weak models). - Planning toggle (an hour). An
update_plan(plan: str)tool, a first-turn instruction to write a plan, and re-inject the plan each turn outside the stored history so compaction never deletes it. - Context manager (a day). The
manage_contextfunction above with tiers T0, T1, T3, T4. Skip T2 recall unless your own logs show the model using it. Use the model’s tokenizer or a close approximation for counting; set a hard ceiling with headroom for the output budget. - Stuck detector (an hour). Hash (tool name, args) per call; nudge at 5 identical in a row; stop at 8 identical failures.
- Eval harness (a day). 50-100 of your real tasks in containers with automated checks. Run each config once per task. Paired McNemar on pass/fail plus mean cost per task. Optionally, a judge prompt that labels each turn’s phase.
The genuinely hard parts: (a) a reliable pass/fail verifier for your own tasks, which is where most of the effort goes; (b) choosing what counts as a “bulky” observation and writing stubs that keep enough for the model to know what it threw away (path, command, exit code, first and last lines). Reach for: LangGraph or your own loop, Docker, ruff, tiktoken or the provider’s token counter, statsmodels for McNemar (statsmodels.stats.contingency_tables.mcnemar(exact=True)).
How to Improve It
- Make the knobs adaptive per task. The paper shows the right setting depends on model and task type. A cheap probe at turn 1 (does the model emit well-formed bash? is the task shell-centric?) could pick the tool set per task. Testable: does an adaptive picker beat the best fixed config on a mixed SWE + Terminal suite?
- Fix recall rather than abandon it. Recall failed because models do not know what they lost. Better stubs (“elided 340 lines of
pytest -xoutput, 3 failures in test_io.py”) or automatic re-hydration when the model touches the same file path could turn M2 from dead weight into a win. Testable by comparing T2 stub variants at 32k on the 30B model, where recall is at least used. - A planning protocol for strong models. Planning helped strong models mostly by giving them a stopping signal. A tiny “definition of done” tool or a single verification checkpoint might capture that cost saving without the plan overhead. Testable: 550B with
declare_done(criteria)vs full planning vs none. - Run the factorial the authors could not. Planning x tools x context tier x window, at least for one strong and one weak model. The interesting unknown is whether planning matters more when summaries are destroying task state at small windows.
- Add frontier and tool-schema-trained models. The tools-vs-bash crossover may vanish or reverse for models trained hard on a specific edit-tool schema. Repeat the 128k ablation on two closed coding models, plus a non-Python benchmark, before you trust the crossover for your stack.
Glossary
- Harness — the software around a model that runs the loop, exposes tools, manages context, and enforces safety.
- ReAct loop — repeat: the model reasons, picks an action (tool call), sees the result.
- Context window budget — the maximum tokens the model can see at once; here artificially set to 32k, 64k, 96k, or 128k.
- Context overflow — history exceeds the window, so the run cannot continue (T0’s main failure).
- Elision (M1) — replacing an old tool output with a short placeholder.
- Recall (M2) — saving the original before elision and letting the model fetch it back with
recall_event(id). - Summarization (M3) — asking the model, in a separate call, to compress old turns into a short running summary.
- Soft / hard threshold (B1 / B2) — the fill levels (60% / 85%) at which T4 elides and then summarizes.
- Preamble / recent window / middle — the never-touched start, the never-touched last turns, and the compactable region between.
- Action space — the set of tools the model can call; here full typed tools vs bash only.
- Re-patch — editing a file the agent already edited, a sign of incremental trial-and-error.
- SWE-Bench Verified — 500 real GitHub issues in Python repos; success = hidden tests pass.
- Terminal-Bench 2.1 — 89 end-to-end command-line tasks with automated checks.
- McNemar test — a paired test that compares two settings using only the tasks where they disagree.
- Benjamini-Hochberg — a correction that keeps the expected share of false “significant” results near 5% when you run many tests.
- LLM judge — a model prompted to label each turn’s phase (localize, fix, verify…); here checked against humans at 94% agreement.