Agent Harness Engineering: A Survey

TL;DR

When an LLM agent does long-running work — fixing a bug, driving a browser, running a terminal for an hour — success is not just “did the model pick good next tokens.” It is whether the surrounding system fed it the right context, ran its actions safely, remembered what already happened, recovered from errors, and stopped at the right time. The authors call that surrounding system the agent harness and argue it is a distinct engineering object that jointly determines performance with the model.

The paper’s spine is ETCLOVG, a seven-layer taxonomy: Execution environment, Tool interface, Context/memory, Lifecycle/orchestration, Observability, Verification, and Governance. They map 170+ open-source projects onto these layers to show where the ecosystem is dense (execution, tools, orchestration, evals) and where it is thin (observability, governance). The headline evidence: swapping only the harness — not the model weights — moved one model up to 10x on coding benchmarks, and lifted a fixed GPT-5.2-Codex agent from 52.8% to 66.5% on Terminal-Bench 2.0. This is a survey, not a new method, so its value is a shared vocabulary and a checklist for anyone building or selling agent systems: it tells you the seven surfaces you must design, the three tradeoffs that bind them together, and where the open problems are.

Problem & Motivation

Here is the pain in one sentence: the same model scores wildly differently depending on the code wrapped around it, and nobody has a shared name or map for that code.

Two concrete symptoms:

• The vocabulary is fragmented. Academic papers study “planning,” “memory,” “tool use,” “verification,” and “safety” as separate research topics. Production teams at OpenAI and Anthropic talk about “context pipelines,” “tool contracts,” “execution policies,” “traces,” “retries,” “handoffs,” and “permission boundaries.” These are often the same problems under different names, so a designer cannot look up “what are all the parts I need to build” in one place. Three questions had no clean answer: where does the harness begin and end, how should its interacting parts be organized, and which parts of the design space are already well covered.

• The harness measurably moves the score, and people keep crediting the model. If you report “GPT-5 gets 66% on this benchmark,” that number is a property of the model-plus-harness pair, not the model. Change the edit-tool format or inject context through middleware and the score jumps. So benchmark comparisons that hold the harness fixed (or vary it as an experiment) are the only honest ones — and most published comparisons do neither.

Prior work fell short because it was either (a) a component view — Meng et al.’s six-part “execution loop, tool registry, context manager, state store, lifecycle hooks, evaluation interface,” which describes one harness but not the ecosystem — or (b) scattered single-topic surveys (a memory survey here, a tool-use survey there) that never connect into a systems-level whole. This paper’s bet is that harness engineering is one integrated systems problem, and you lose the plot if you study the layers in isolation.

What’s New (Core Contribution)

Three contributions, each “before X / now Y”:

1. The harness as a first-class system object.

   • Before: the “agent” was the model plus some glue code treated as an implementation detail; performance was attributed to the model.
   • Now: the harness is defined as the engineered wrapper that turns model calls into “bounded, stateful, tool-mediated task execution,” and its design is treated as a measurable experimental factor. The paper frames the model + harness as a closed-loop control system — the model is the controller’s policy, the harness is the controller CH that shapes what the model sees, what its outputs do, and when the loop stops.

2. The ETCLOVG seven-layer taxonomy.

   • Before: component lists (Meng et al.’s six parts) or topic silos.
   • Now: seven layers organized by engineering surface rather than runtime role, which lets you map complete harnesses, protocols, sandboxes, memory systems, eval frameworks, and governance modules onto the same grid. Two deliberate design moves: Observability (O) is promoted to its own layer (it has its own tool ecosystem — Langfuse, Phoenix, OpenTelemetry), and Governance (G) is made first-class (spanning model-level, system-level, and org-level controls). State management is filed under Lifecycle, where it is created and consumed.

3. A systematic ecosystem map of 170+ projects + a cross-layer synthesis.

   • Before: no reproducible census of what exists.
   • Now: a documented collection protocol (GitHub search, papers, curated lists, company blogs, frozen May 2026) with inclusion criteria, each project coded to ETCLOVG layers. On top of the map, three recurring “binding constraints” that no single layer resolves: the cost-quality-speed trilemma, the capability-control tradeoff, and the harness coupling problem.

Be precise about what is genuinely new: the individual techniques (ReAct, MCP, sandboxes, LLM-as-judge) are all prior work. What is new is the unifying frame and the map — naming the object, drawing its boundary, and showing the ecosystem’s shape. This is a cartography contribution, not an algorithm.

How It Works (Technically)

Because this is a survey, the “mechanism” is the taxonomy plus the control-loop frame that binds it. I’ll (1) demystify the one bit of formalism, (2) walk one task through all seven layers, and (3) give simplified pseudocode of the harness loop that is the paper’s mental model.

The one equation, in plain English

The paper’s conceptual claim can be written as:

measured_behavior = f( model M, harness controller CH, task, environment )

There is no heavy math in this paper — that is the point. The formal move is only this: treat the model as a fixed policy and treat the harness as a controller wrapped around it in a feedback loop. In control-systems language, a controller reads the state, decides an action, applies it, observes the result, and repeats. The harness is that controller: it builds the prompt (what state the model sees), routes the model’s output to a tool (turns a decision into an action), captures the result (observation), and decides whether to continue or stop.

What this does operationally: it explains why you cannot attribute a benchmark score to M alone. Changing CH — a different context policy, a new tool schema, an added verifier, a retry loop — changes the controller and therefore changes the measured behavior of the same M. That is the whole argument for “harness engineering matters,” made rigorous by refusing to treat CH as a constant.

Three tradeoffs fall directly out of this frame (these are the paper’s real analytical payload):

• Cost-quality-speed trilemma. Stronger sandboxes, richer memory, deeper evals, more tracing — each improves quality or safety but adds latency, tokens, or dollars. You cannot maximize all three; you choose which checks run synchronously (inline, slow, safe), which run async (offline regression), and which failures justify expensive recovery.
• Capability-control tradeoff. Every capability you hand the agent (a bigger tool menu, persistent memory, a permissive sandbox) expands task coverage and the blast radius when it goes wrong. More authority = more control problem.
• Harness coupling problem. The layers are not independent. The execution environment changes eval results (package availability, reset semantics). Tool descriptions eat context budget. Observability traces only become governance evidence if identity/permission state is captured at the same granularity. So a change that looks good in isolation can degrade the whole rollout. Test harness changes as system changes, not unit changes.

The seven layers (ETCLOVG), one line of purpose each

• E — Execution & Sandbox: where the agent’s actions physically run. Not just security: also reproducibility (destroy-and-rebuild a container to reset state for evals/training) and liveness (a sandbox is a “license” to act freely inside a boundary, so you stop asking the human to approve every file write). Anthropic reports sandboxing cut Claude Code’s permission prompts by 84% while keeping safety. Categories run from managed cloud sandboxes (E2B, Daytona, Modal) through computer-use VMs, WebAssembly code sandboxes, to OS-level permission jails (bubblewrap, Seatbelt). Trend: from shared-kernel containers toward dedicated-kernel microVMs, because LLM-generated code has unpredictable syscalls.
• T — Tool interface & protocol: how capabilities are described, discovered, and invoked. The fault line: more tools = more coverage but worse decisions and more tokens. Standards are organized by the boundary they cross: Function-calling (Model↔Function), MCP (Agent↔Capability), A2A (Agent↔Agent), AGENTS.md (Agent↔Repo). MCP is winning on “ecosystem liquidity” — reuse a growing server catalog instead of writing connectors.
• C — Context & memory: what the model can see across short-term (active window: compaction, tool-result clearing, cache-aware ordering), mid-term (session state across runs), and long-term (persistent memory stores). Key insight: this is state estimation, not “cram more tokens in.” Longer context does not equal better task-state tracking (“context rot”).
• L — Lifecycle & orchestration: the control flow that reads/writes state. Three scales: single-agent inner loop (ReAct-style: Claude Code, Aider, Codex), multi-agent patterns (hierarchical, team, workflow, fan-out, graph — AutoGen, LangGraph, DeepAgents), and full issue→PR pipelines (Vibe Kanban, Symphony). Execution is stateless-replay, stateful, or hybrid.
• O — Observability & operations: traces, cost, failures, reliability. Structured trace trees (Langfuse, Phoenix) built on OpenTelemetry’s GenAI conventions; cost tracking (Helicone, TensorZero) and cost optimization (FrugalGPT’s cascade, GPTCache’s semantic cache); reliability engineering (checkpoint/resume, retry, failure taxonomies like MAST’s 14 modes).
• V — Verification & evaluation: turning tasks and traces into judgement and regression feedback. Modeled as a five-stage task-to-feedback lifecycle: (1) task/benchmark grounding, (2) pre-execution readiness validation, (3) controlled execution + trace capture, (4) multi-level judgement + failure attribution, (5) continuous regression + deployment feedback. The heart: a failed run may be the harness’s fault (broken tool, stale context, non-reset sandbox, flaky test, unstable judge), not the model’s.
• G — Governance & security: permissions/identity, lifecycle hooks, component hardening, declarative “constitutions” (YAML policy files), and audit logs. The survey shows this layer is sparsely implemented in real systems — information-flow control, identity management, and formal verification are absent from every one of six audited agents.

Architecture & data flow

flowchart LR
  subgraph Core["Structural core"]
    E["E · Execution/Sandbox<br/>where actions run"]
    T["T · Tool interface<br/>how capabilities are called"]
    C["C · Context/Memory<br/>what the model sees"]
    L["L · Lifecycle/Orchestration<br/>control flow + state"]
  end
  subgraph Plane["Control plane"]
    O["O · Observability<br/>traces, cost, failures"]
    V["V · Verification<br/>eval + failure attribution"]
    G["G · Governance<br/>permissions, policy, audit"]
  end
  M["LLM (fixed policy)"]
  L -->|builds prompt| C
  C -->|context window| M
  M -->|proposed action| T
  T -->|runs in| E
  E -->|observation/result| L
  O -.->|monitors all| Core
  V -.->|scores traces| O
  G -.->|constrains all| Core

Schematic of the closed-loop harness controller: watch one task cycle through observe → build context → model decides → tool runs in the sandbox → result returns, with the O/V/G control plane watching. Illustrative, not the paper's measured data.

One task, traced through all seven layers

Take a coding agent asked to “fix the failing test in this repo and open a PR.”

1. E: the harness spins up a fresh microVM with the repo checked out (reproducible baseline; can be reset if the run corrupts state).
2. L: an initializer step decomposes the task into a feature list and writes a progress file + init script (Anthropic’s pattern to stop the agent one-shotting the whole task).
3. C: the harness assembles the context window — the failing test output, relevant files, and a compacted history — clearing stale tool results to fight context rot.
4. T: the model proposes run_tests() then edit_file(...); the harness validates the calls against tool schemas (MCP/function-calling) and routes them.
5. E again: the edit and test run inside the sandbox; no human approval prompt because the sandbox is the license boundary.
6. O: every LLM call, tool call, cost, and exception is captured as a span tree for later inspection.
7. V: a verifier checks whether tests actually pass (not just whether the agent says they pass — agents over-praise their own work), and attributes any failure to model vs. tool vs. environment.
8. G: before the agent pushes, a governance hook checks it is allowed to write to the branch and logs the action to an audit trail; a human approves the final PR.

Output: a validated PR, plus a trace that can be turned into a regression test if it failed. Every layer touched the run; remove any one and reliability drops.

The algorithm, simplified

There is no single algorithm to reproduce — the “algorithm” is the harness control loop the whole paper describes. Here is that loop with the seven layers made explicit, so you can see where each ETCLOVG surface plugs in:

# One agent task = one closed-loop run of the harness controller CH around a fixed model M.
def run_harness(task, model):
    sandbox = provision_sandbox(task)          # E: reproducible, isolated "license to act"
    state   = init_lifecycle(task, sandbox)    # L: feature list, progress file, checkpoints
    memory  = MemoryStore()                    # C: short/mid/long-term, actively managed

    for step in range(MAX_STEPS):
        # C: assemble what the model sees this step (compact, clear stale results, order for cache)
        context = build_context(task, state, memory.read(task))

        # M is a FIXED policy; CH decides everything around it
        action = model(context)                # proposed tool call / next action

        if not governance_allows(action, state):   # G: permission/identity/policy hooks (inline)
            action = escalate_to_human(action)      # G: human-in-the-loop for risky actions

        result = run_in_sandbox(action, sandbox)    # E+T: validated tool call executes safely
        trace.record(step, action, result)          # O: span tree — cost, latency, exceptions

        state  = update_lifecycle(state, action, result)  # L: persist artifacts, checkpoint
        memory.write(summarize(result))                   # C: write-manage-read memory loop

        verdict = verify(state, result)          # V: did it ACTUALLY work? attribute failure
        if verdict.done:
            break
        if verdict.stuck:
            state = recover(state)               # O/L: retry, reset, or re-decompose

    feedback_to_regression(trace, verdict)       # V: turn anomalous traces into future tests
    return state.output

The teaching point: M appears on exactly one line. Everything else is CH. That is the paper’s thesis rendered as code — most of an agent’s reliability is decided by the code that is not the model call.

Built on Prior Work

The paper is a synthesis, so its “lineage” is a map of the field it is organizing. What it borrows and what it adds:

The intellectual debt is heaviest to the practitioner blogs (Anthropic, OpenAI, LangChain) — the survey’s real move is treating those engineering reports as primary sources and giving them a research-grade structure.

Results & Evidence

This is a survey, so “results” = the map’s findings and the evidence marshaled for the central claim. What the evidence does establish:

• The harness measurably moves scores (strongest evidence). Bölük changed only the edit-tool format/harness across 15 models and got up to 10x improvement on coding for one model. LangChain’s DeepAgents (Trivedy) lifted a fixed GPT-5.2-Codex from 52.8% → 66.5% on Terminal-Bench 2.0 via system-prompt restructuring, middleware context injection, and self-verification hooks. Meta-Harness (Lee et al.) shows parts of the harness can be automatically optimized to beat hand-built scaffolds.
• Infrastructure noise is real. Anthropic found infrastructure configuration alone shifts benchmark scores by 6 percentage points (p < 0.01) — meaning a chunk of published “model” differences may be plumbing.
• The ecosystem is uneven (the map’s core finding). E, T, L, V are densely covered (coding/web/terminal agents all need runnable environments, tool contracts, loops, and repeatable evals). C appears everywhere but is usually embedded in frameworks, not released standalone. O and G are thin in open source — operational control matured later than runtime and benchmark infra.
• Governance is a bottleneck. Across six audited agents (Codex, Gemini CLI, OpenHands, Browser Use, Nanobrowser, Skyvern), no agent implements all defense categories; information-flow control, identity management, and formal verification are absent from every one.
• The observability-eval gap. A LangChain survey found 89% of teams use observability but only 52% run offline evals — teams can see what agents did without judging whether it was correct.

What the evidence does NOT establish (the authors are refreshingly honest):

• “Harness > model” is not proven in general. The strongest controlled evidence is all from coding-agent benchmarks, which are overrepresented because they have unusually rich public traces. The claim is only “performance depends jointly on model and harness,” not that the harness dominates everywhere.
• The corpus is a map, not a census. It is biased toward English, GitHub-visible, open-source projects whose maintainers publish internals. Commercial systems are underrepresented; “absent from a layer” means “not publicly evidenced,” not “not implemented.”
• No inter-annotator reliability. Coding used a single-primary-coder protocol with author audit, so no Cohen’s kappa is reported. Layer assignments are one team’s judgement.
• The taxonomy is descriptive, not normative. ETCLOVG classifies harnesses; it does not yet tell you how to design one. Turning it into design guidance is explicitly future work.

A striking datapoint on the “harness-as-assumption” principle: when Anthropic upgraded Opus 4.5 → 4.6, they removed the sprint construct and context resets, cutting cost from $200 → $125 while keeping quality. Every harness component encodes an assumption about what the model cannot do — and those assumptions go stale as models improve.

How You’d Use It

For an AI-services company, this survey is most valuable as a capability checklist and a diagnostic frame. Concrete uses:

• A build vs. buy grid for client engagements. When scoping an agent project, walk ETCLOVG as seven line items. For each: build, adopt open source, or buy commercial? E → E2B/Daytona vs. self-hosted microVMs. T → MCP servers. O → Langfuse/Phoenix. V → promptfoo/DeepEval. This turns “build an agent” into a bounded seven-part statement of work you can price.
• A reliability diagnosis when a client’s agent “just isn’t reliable.” The paper’s failure-attribution frame says: don’t blame the model first. Check the harness layers in order — is the sandbox resetting? Is context rotting past ~100 turns? Are tool schemas bloated? Is the verifier actually verifying or trusting the agent’s self-report? Anthropic’s four failure modes (one-shotting, premature completion, broken handoff state, marking done without testing) are a ready-made checklist.
• A moat argument for productized harnesses. The “frameworks → platforms” shift (durable workspaces, identity, billing, observability, governance, human handoff) is exactly the layer where a services firm can build reusable IP. Your differentiated asset is the harness, not the model — and it stays valuable across model upgrades if you build in the “harness-as-assumption” discipline to prune stale scaffolding.
• Cost control as a sellable service. The O-layer techniques (semantic caching à la GPTCache, model cascading à la FrugalGPT’s up-to-98% savings, budget-aware routing) are concrete, measurable line items you can install and bill against. Pair with the warning that cutting resources to save cost can silently degrade quality (the 6pp infrastructure-noise finding).
• A governance/compliance offering. Since G is the thinnest layer in the wild and enterprises need it, “we add permission models, audit logging, and a policy constitution to your agent” is an under-supplied, high-trust service. Table 3’s mapping of governance mechanisms to risk categories is a starting audit template.

Where it slots in: this is the reference you read before architecting a client’s agent, and the frame you use to write the SOW and the post-mortem. It replaces “we’ll build an agent” with “we’ll build and instrument these seven surfaces.”

Build Your Own (Minimal Recipe)

You cannot “reimplement a survey,” but you can build the minimal ETCLOVG-complete harness the paper describes — the smallest thing that captures ~80% of the value. Build order (each maps to a layer):

1. L — the loop first. A while loop: build context → call model → parse a tool call → run it → append result → check stop condition. ~50 lines. This is your controller CH.
2. E — a sandbox. Start with an off-the-shelf managed sandbox (E2B or Docker) so tool actions can’t touch the host. Get reset-to-baseline working early; you need it for evals.
3. T — tool schemas. Use function-calling/MCP JSON schemas. Keep the menu small (the paper’s repeated warning: oversized tool menus degrade reliability). Validate every call before executing.
4. C — context management. Add compaction and tool-result clearing so runs survive past a few dozen turns. Externalize durable state (a progress file, the repo) rather than trusting the prompt.
5. O — tracing. Wrap every model/tool call with an OpenTelemetry-style span (or just Langfuse’s @traceable). This is cheap and pays off immediately in debugging.
6. V — a verifier + a handful of eval tasks. Separate “the agent claims done” from “a check confirms done.” Even 10 real tasks with automated pass/fail beats vibes.
7. G — one governance hook + an audit log. A single permission check before risky actions and an append-only log of what the agent did.

The one or two genuinely hard parts:

• Context/state fidelity over long horizons (C + L). Every summarize/retrieve/compact step can silently delete a constraint or preserve a stale assumption. Keeping the agent’s working state aligned with the true task state is the deep unsolved problem — the paper reframes it as “state estimation,” and there is no library that solves it for you.
• Honest verification (V). Building a verifier the agent can’t game (and that attributes failure to the right layer) is harder than it looks, because agents over-praise their own work and infrastructure noise masquerades as model failure.

Libraries/models to reach for: LangGraph (graph orchestration for L), E2B/Daytona/Docker (E), MCP servers (T), Langfuse or Arize Phoenix (O), promptfoo / DeepEval / lm-evaluation-harness (V). A capable coding model (Claude/GPT-class) as the policy. You can stand up a credible v1 in days; the reliability polish on C and V is where the months go.

How to Improve It

Limitations of the survey and of the field it maps, turned into testable directions:

1. Make ETCLOVG normative, not just descriptive. The taxonomy classifies but doesn’t guide. Build a decision procedure: given a task profile (risk, horizon length, tool count), recommend an isolation primitive, a memory policy, and a verification depth. Testable: does following the recommendation beat a naive baseline on Terminal-Bench?
2. Trace-native evaluation to close the 89%/52% gap. Automatically convert anomalous production traces into regression tests, compute trajectory metrics directly over spans, and feed diagnostics back into prompt/tool/context changes. Reflexion showed agents can learn from their own traces short-horizon; extend it to multi-session harnesses.
3. A “harness-as-assumption” auto-simplifier. Since components go stale as models improve (the $200→$125 example), build a meta-monitor that tracks which interventions (context resets, evaluator loops, tool restrictions) are still load-bearing and proposes removing the rest on each model upgrade. This is directly monetizable.
4. A unified governance benchmark. No benchmark today evaluates a complete governance stack under one adversary model. Build one that reports defense effectiveness, false-positive rate, and overhead jointly — because a guardrail that blocks 99% of attacks but also 30% of legit actions is useless.
5. Cross-layer coupling tests. The coupling problem says local optimizations can hurt the whole. Build a harness A/B harness that changes one layer at a time and measures system effect, so “this new tool helped in isolation but tanked the rollout” becomes visible. Extend the corpus beyond coding agents to test whether the whole frame generalizes.

Glossary

• Agent harness — the engineered wrapper around an LLM that turns model calls into bounded, stateful, tool-using task execution (sandbox + tools + context + control loop + tracing + evals + guardrails).
• ETCLOVG — the paper’s seven-layer taxonomy: Execution, Tooling, Context, Lifecycle, Observability, Verification, Governance.
• Closed-loop control — a system that observes state, acts, observes the result, and repeats; the harness is the controller wrapped around the model.
• Controller (CH) — the harness treated as the thing that decides what the model sees, what its output does, and when to stop; distinct from the model M.
• Sandbox — an isolated environment where agent actions run safely; serves security, reproducibility (resettable), and liveness (a “license” to act without per-action approval).
• microVM — a lightweight virtual machine with its own kernel (Firecracker, Kata); stronger isolation than a container, used because LLM code has unpredictable syscalls.
• MCP (Model Context Protocol) — Anthropic’s JSON-RPC standard for exposing tools/resources to an agent via reusable servers.
• A2A (Agent-to-Agent) — Google’s standard for communication and delegation between agents.
• Function calling — the model emitting a structured (JSON) call to a named tool, with defined arguments and return.
• Context engineering — actively managing the model’s input window (which docs/memories are included, how ordered, filtered, summarized) rather than treating it as a passive transcript.
• Context rot / drift — the degradation of an agent’s task-state tracking as context grows or is repeatedly compressed; longer context does not mean better tracking.
• Compaction — summarizing or pruning the context window to fit the token budget while trying to preserve constraints.
• Stateless replay vs. stateful execution — reconstructing a run from recorded history vs. persisting operational state (files, DBs, task graphs) outside the prompt; hybrid does both.
• ReAct — the interleave-reasoning-and-acting agent loop; the basic single-agent primitive.
• Orchestration patterns — hierarchical (a controller assigns work), team (named specialist agents), workflow (explicit stages), fan-out (parallel exploration), graph (nodes = agents/tools/states).
• Observability — collecting traces, costs, and failure signals; here a trace is a tree of “spans,” one per LLM/tool/retrieval call.
• OpenTelemetry (OTel) — the de facto tracing standard; its GenAI semantic conventions define span attributes for model, tokens, latency.
• LLM-as-judge — using an LLM to score another agent’s output or trace; powerful but an unstable evaluator.
• Failure attribution — deciding whether a failed run was caused by the model, a tool, the sandbox, stale context, a flaky test, or the judge.
• Governance / constitution — runtime constraints on agent behavior; a “constitution” is often a declarative YAML policy file specifying permissions and limits.
• Human-in-the-loop — routing risky actions to a person for approval.
• Cost cascade / semantic cache — routing easy queries to cheaper models (FrugalGPT) and reusing cached answers for paraphrased queries (GPTCache) to cut token cost.
• Cost-quality-speed trilemma — you cannot maximize safety/quality, latency, and dollar-cost simultaneously; you choose which checks are synchronous.
• Capability-control tradeoff — every capability granted to the agent expands both what it can do and the blast radius when it misbehaves.
• Harness coupling — the layers interact, so a change good in isolation can degrade the whole system; test harness changes as system changes.