Meta-Harness: End-to-End Optimization of Model Harnesses

TL;DR

The code wrapped around an LLM — what it stores, retrieves, and shows the model at each step — can swing performance by 6x on the same benchmark, but people still build that code by hand. Existing “text optimizer” tools that automate prompt/artifact improvement (OPRO, TextGrad, GEPA, AlphaEvolve) all compress their feedback down to a scalar score, a short summary, or the single most recent attempt, because that’s what fits in a prompt. Meta-Harness instead gives its proposer — a coding agent, specifically Claude Code — full filesystem access to every past harness it has tried: the code, the score, and the complete raw execution trace. The agent greps and cats its way through this archive the way a human engineer would, forms a hypothesis about what’s actually broken, and rewrites the harness. Across text classification, retrieval-augmented math, and agentic coding, harnesses discovered this way beat the best hand-built baselines and match or exceed prior automated text optimizers using a fraction of the search budget.

Problem & Motivation

Two facts set up the problem. First, the harness matters enormously: swap the scaffolding code around a fixed model and you can get a 6x performance gap on identical tasks, because the harness decides what context the model sees, when it retrieves something, and how state carries across steps. Second, harness engineering today is almost entirely manual — a practitioner reads failures, tweaks a retrieval rule or a prompt template, reruns, and repeats.

The obvious fix is to treat this like any other text-optimization problem and throw an existing automated optimizer at it (OPRO, TextGrad, GEPA, AlphaEvolve/OpenEvolve, Feedback Descent, TTT-Discover). The paper argues these tools are the wrong shape for this job. Every one of them was designed around a tight feedback budget: some only condition on the current candidate (memoryless), some only see a scalar score, some see an LLM-written summary or a short template. That’s a reasonable design choice when you’re tuning a single prompt for a single LLM call — one call, one output, one score, easy to summarize. A harness isn’t that. It’s a stateful program: a decision made at step 1 (what to store in memory) can silently break behavior at step 40, and diagnosing that requires tracing through the actual execution, not a compressed summary of it. Table 1 in the paper quantifies the gap: prior text optimizers work with roughly 100–30,000 tokens of feedback per optimization step. A single harness evaluation in this paper’s settings can produce up to 10,000,000 tokens of diagnostic information — about three orders of magnitude more. Compressing that down to fit a prompt throws away exactly the information needed to trace a failure back to its cause.

What’s New (Core Contribution)

• Full raw history via filesystem, not a compressed summary window. Before: text optimizers keep the last candidate, a fixed-size window, or an LLM-generated summary (Table 1’s “History” column: Last / Window / Summary). Now: every evaluated harness gets its own directory with source code, scores, and complete execution traces (prompts, tool calls, model outputs, state updates), and the proposer queries this archive with ordinary shell tools (grep, cat) rather than having it stuffed into one prompt.
• A coding agent as the proposer, not a raw LLM completion. Before: text optimizers call an LLM once per step with a fixed prompt template built by the outer loop. Now: the proposer is an agentic system (Claude Code) that decides what to inspect — it can open any prior harness’s code or trace, run commands, and validate its own edits by interacting with the codebase, rather than reacting only to whatever the outer loop chose to show it.
• A deliberately minimal outer loop. Before: methods like AlphaEvolve/OpenEvolve impose structure — a program database, tournament parent-selection, fixed mutation operators. Now: Meta-Harness has no parent-selection rule at all; the proposer is free to inspect any prior harness and either make a local edit or a full rewrite. The authors argue this is intentional: coding models already have a bias toward coherent, reusable programs rather than brittle hacks, so representing harnesses as programs is itself a form of regularization, and hand-designed search heuristics become unnecessary scaffolding that will only get in the way as coding agents get better.
• Demonstrated end-to-end across three genuinely different domains, with evidence of transfer: the text-classification harness generalizes to 9 unseen datasets, and a single math-retrieval harness improves accuracy across 5 held-out models it never saw during search.

How It Works (Technically)

The objective, in plain English. A harness H is a stateful program wrapping a frozen LLM M. For a task x, running H with M produces a trajectory τ (the full sequence of prompts, model outputs, and state updates), and a task-specific reward function scores that trajectory: r(τ, x). The goal is to find the harness H* that maximizes expected reward over the task distribution:

H* = argmax_H E_{x~X, τ~p_M(H,x)} [ r(τ, x) ]

Read operationally: this just says “pick the wrapper code that gets the best average score across the tasks you care about.” When you also care about a second axis — say, accuracy and context-token cost — Meta-Harness doesn’t collapse that into one scalar; it evaluates candidates by Pareto dominance and reports the resulting frontier (a curve of “best accuracy achievable at each context budget”), so a user can pick their own operating point instead of the paper baking in one tradeoff.

The search loop (Algorithm 1). Start with a population of seed harnesses (typically the domain’s existing baselines: zero-shot, few-shot, and the best hand-built system). Evaluate each on a search set (a held-out-from-the-test-set slice of tasks used purely to drive search), and write everything — code, scores, traces — into a filesystem archive D. Then, for N iterations:

1. The proposer queries D: it reads prior harness code, scores, and traces — in the paper’s most demanding setting, a median of 82 files per iteration, touching 20+ prior candidates.
2. The proposer proposes k new harness candidates (edited code, full rewrites, or novel designs).
3. Each candidate first passes a cheap interface-validation check (does it import, instantiate, and run without crashing on a couple of examples?). Only validated candidates get the expensive full evaluation.
4. Validated candidates are evaluated on the search set; their code, scores, and traces are logged into a new directory in D, and the loop repeats.

At the end, the paper reports the Pareto frontier of harnesses found during search, and does one final evaluation on the held-out test set — the proposer never sees test-set results at any point during search, only search-set feedback.

Why code space, not prompt space. Because harnesses are stateful, a bug can be caused by an interaction between two changes made several iterations apart. The paper’s clearest evidence for this is a qualitative trace from the TerminalBench-2 run (Appendix A.2, summarized in Results below): the proposer initially conflates two independent changes, watches both regress, and only on the third iteration explicitly reasons out that one specific change (a prompt rewrite) was the actual cause, isolates it, and reverts course. That kind of causal debugging is only possible with access to the raw trace of what actually happened, not a scalar score saying “this candidate got 58.9%.”

Implementation specifics. Each harness in the experiments is a single Python file. The proposer is Claude Code running Opus-4.6, steered by a short domain-specific “skill” (a natural-language instruction set) that defines where to write new harnesses, how to inspect prior ones, and what files are off-limits — but deliberately does not tell it how to diagnose problems. A typical run evaluates ~60 harnesses over ~20 iterations and completes in a few hours of wall-clock time.

Architecture & data flow

The outer search loop — how one iteration turns prior experience into a new, evaluated candidate:

flowchart TD
  D[(Filesystem archive D<br/>code + scores + traces, one dir per harness)] -->|grep / cat, no single-prompt dump| P[Proposer<br/>Claude Code + Opus-4.6, guided by a short skill]
  P -->|writes k new candidates| H[New harness code]
  H --> V{Interface validation<br/>fast smoke test}
  V -->|fail, cheap to discard| X[Discarded]
  V -->|pass| E[Full evaluation on search-set tasks]
  E --> S[Scores + prompts/tool-calls/outputs/state updates]
  S -->|logged into a new directory| D

What a single evaluated rollout looks like inside one candidate harness (the objective made concrete):

flowchart LR
  X[Task instance x] --> H[Harness H<br/>builds prompt from current state]
  H --> M[Frozen LLM M]
  M --> U[Harness updates its state]
  U -->|next step, if any| H
  U --> T[Trajectory τ<br/>the full run, logged]
  T --> R["Reward r(τ, x)"]

How much feedback each method actually gives its optimizer per step, on a log scale, using the paper's own numbers (Table 1). Meta-Harness's raw-trace archive is roughly three orders of magnitude larger than any prior text optimizer's feedback budget — this gap is the paper's central empirical claim about *why* the other methods are mismatched to harness search.

A schematic animation of the outer loop: the proposer reads a growing archive of prior candidates (source + traces), writes a new candidate, and — if it survives validation — gets evaluated and logged back into the archive for the next iteration to read. Illustrative, not from paper data.

The algorithm, simplified

# One iteration of the Meta-Harness outer loop.
# D is a filesystem: D[harness_id] = {"code": str, "score": float, "trace": str}

def meta_harness_iteration(D, tasks, model, proposer, k=2):
    # 1. Proposer inspects prior experience via filesystem ops, not a single prompt.
    #    It decides what to read -- source code, traces, or both -- for any prior candidate.
    context = proposer.inspect(D)  # e.g. grep for failure patterns, cat specific traces

    # 2. Proposer writes k new harness candidates, informed by what it just read.
    candidates = proposer.propose_harnesses(context, k=k)

    for harness_code in candidates:
        # 3. Cheap smoke test before paying for a full evaluation.
        if not passes_interface_validation(harness_code):
            continue  # discarded near-zero cost

        # 4. Full evaluation on the search set (NOT the held-out test set).
        score, trace = evaluate(harness_code, model, tasks.search_set)

        # 5. Log everything -- code, score, AND raw trace -- into the archive.
        D[new_id()] = {"code": harness_code, "score": score, "trace": trace}

    return D  # grows by up to k entries; repeat for N iterations, then take the
              # Pareto frontier over (score, cost) and do one held-out test eval

Built on Prior Work

Results & Evidence

Online text classification (GPT-OSS-120B classifying LawBench/Symptom2Disease/USPTO-50k examples one at a time, updating memory as it goes): the discovered harness (a label-primed, query-anchored retrieval prompt) reaches 48.6% average accuracy, beating ACE (Agentic Context Engineering, a strong hand-built reflective-memory system) by 7.7 points and MCE (evolving skill library) by 8.6 points — while using only 11.4K context tokens versus ACE’s 50.8K and MCE’s 28.5K. Against other automated optimizers under a matched evaluation budget, Meta-Harness matches OpenEvolve’s and TTT-Discover’s final accuracy after just 4 evaluations (they need ~60), and its eventual best accuracy is more than 10 points higher than either (Table 4: Meta-Harness best 56.7 vs. OpenEvolve 45.6 vs. TTT-Discover 43.3 vs. GEPA 40.2). Because the search optimizes accuracy and context cost jointly, it also produces a full Pareto frontier of harnesses trading one for the other (Figure 3 / Table 9), not a single fixed operating point.

The ablation that matters most (Table 3). Holding everything else fixed, giving the proposer only scalar scores reaches 41.3 best accuracy; adding LLM-written summaries on top barely moves that (38.7 best — summaries can hurt, because they compress away the exact detail needed to diagnose a failure); giving it the full raw-trace filesystem reaches 56.7 best, and Meta-Harness’s median candidate (50.0) beats the best candidate found under either compressed condition. This is the paper’s strongest piece of evidence that raw trace access, not just “more search,” is what’s doing the work.

Out-of-distribution generalization: the selected text-classification harness, evaluated on 9 entirely new datasets never touched during search, still beats ACE on average (73.1% vs. 70.2%) and wins on 6/9 individual datasets — evidence it learned a generally useful strategy rather than overfitting to the three search datasets.

Retrieval-augmented math reasoning: a single harness, discovered over 40 iterations against GPT-OSS-20B, is a compact 4-route BM25 program (separate retrieval policies for combinatorics, geometry, number theory, and a default route, chosen by lightweight keyword/regex rules). Evaluated on 200 held-out IMO-level problems across 5 models — including 4 it never saw during search (GPT-5.4-nano/mini, Gemini-3.1-Flash-Lite, Gemini-3-Flash) — it improves accuracy by 4.7 points on average over no retrieval, and modestly beats a strong fixed BM25 baseline (+1.3 points) while avoiding the regressions that dense retrieval and random few-shot prompting sometimes cause on individual models.

Agentic coding (TerminalBench-2, 89 long-horizon terminal tasks). On Claude Opus 4.6, the discovered harness scores 76.4% pass rate, beating the hand-engineered Terminus-KIRA (74.7%) and ranking #2 on the public leaderboard among all Opus 4.6 agents (only ForgeCode’s 81.8% is higher — and the authors note they could not reproduce that number from ForgeCode’s public code, a useful reminder to treat single leaderboard numbers skeptically). On the weaker Claude Haiku 4.5, the gain is larger: 37.6% vs. the next-best reported agent (Goose, 35.5%), ranking #1 among Haiku 4.5 agents. The entire discovered change is small and additive: a single shell command that snapshots the sandbox environment (OS, installed languages, package managers, /app contents, available memory) and injects it into the very first prompt — about 80 lines added on top of Terminus-KIRA. Per-task analysis shows the gain concentrates on 7/89 tasks that need non-obvious domain tooling (bioinformatics libraries, rendering pipelines, chess engines) — tasks where an agent otherwise burns its first several turns just discovering what’s installed.

Why the winning fix is boring, and that’s the point (Appendix A.2). The TerminalBench-2 search log shows the proposer trying six consecutive structural fixes to the completion/verification logic across iterations 1–6, all of which regress performance, because each one was bundled with a prompt-template change. On iteration 3, the proposer explicitly identifies that the prompt changes, not the structural bugfixes, were the common cause of the regressions — a causal diagnosis it could only make by comparing raw traces across multiple failed candidates. It then pivots to a purely additive change (the env-bootstrap) that avoids touching the fragile completion machinery entirely, which becomes the winning candidate. Later iterations compose that fix with an earlier isolated one and even reference a finding from a separate search run (“not cleaning up service artifacts was worth +18pp”). The authors read this as evidence of genuine hypothesis-driven debugging rather than random mutation, and it’s a legitimately interesting research artifact worth reading in the paper’s appendix.

What the evidence does not establish. The TerminalBench-2 experiment searches and evaluates on the same 89-task benchmark — there is no held-out split, which the authors acknowledge and justify on cost/scarcity grounds, backed by a manual/regex leak audit, but it’s still a discovery result on a contested public benchmark, not a clean generalization claim the way the OOD text-classification and cross-model math results are. The paper uses exactly one proposer configuration throughout (Claude Code + Opus-4.6) and explicitly flags that a broader study across different coding-agent proposers is future work — so it’s unclear how much of the gain is “the method” versus “how good the underlying coding agent already is.” There’s no dollar-cost-normalized comparison against the text optimizers it beats — Meta-Harness generates roughly 1,000x more diagnostic tokens per evaluation (Table 1), so its wall-clock and compute cost per iteration is almost certainly higher, even if it needs fewer iterations overall. And the math-corpus decontamination relies on exact-prefix matching plus fuzzy Jaccard similarity (threshold 0.8) — a reasonable but imperfect filter, not a guarantee against any leakage.

How You’d Use It

Your harness. This is the paper to point at your own agent runtime — the RAG pipeline, memory system, or tool-use loop wrapping whatever model you’re running. Instead of hand-tuning prompts and retrieval rules by feel, set up the same loop yourself: a baseline harness, a search set drawn from your real task distribution, a proposer (a coding agent) with filesystem access to every past attempt’s code, score, and trace, and a cheap validation gate. What it costs to stand up is mostly the logging convention (one directory per candidate: code, score, trace) and a fast smoke test before full evals — not a new framework. What has to be true for it to pay off: your baseline has to actually fail on some slice of your eval set (if it already saturates, there’s nothing to search for), and you need an eval you trust enough to optimize against.

Your automations and business processes. Treat harness search as a pre-ship step rather than a one-off tuning exercise: before shipping a new version of a retrieval policy, a memory system, or a sub-agent’s context-construction logic in a multi-agent system, run a short search against your own eval set and keep the winning harness plus its trace archive as the documented reason for the change. That turns “we tweaked some prompts” into a diffable, auditable artifact you can point to later, instead of engineering hours nobody can reconstruct.

Your workflows and methodologies. The paper’s real finding — that a coding agent debugging from full raw traces beats compressed summaries or scalar scores by a wide margin — generalizes past harness search. Any time you use an agent to iterate on its own output (a prompt, a config, a pipeline step) and you feed it a summary of what went wrong instead of the actual logs, you’re likely leaving performance on the table; the ablation here (56.7 vs. 41.3 best accuracy, full traces vs. scalar scores) is a strong argument for always giving your optimizing agent the raw trace, not a digest of it.

Build Your Own (Minimal Recipe)

Components, in build order:

1. A baseline harness and a search set that’s actually hard for it. Start with the simplest version you already have (zero-shot or few-shot). If the baseline already saturates your eval set, there’s nothing for search to find — filter or curate a set the baseline gets meaningfully wrong. The paper keeps this small on purpose (50–250 examples, enough for ~50 full evaluations per run) because a fast, discriminative eval beats a large slow one.
2. A logging convention the proposer can navigate with grep/cat. One directory per candidate: runs/<id>/{harness.py, score.json, trace.jsonl}. Machine-readable formats and consistent naming matter more than they sound like they would — this is the actual interface the proposer works against.
3. A separate evaluation script, run outside the proposer, that scores a candidate against the search set and writes the log. Don’t make the proposer run its own evals — it’s a distraction from what it’s actually good at.
4. A cheap interface-validation gate: import the module, instantiate the class, run it on 2–3 toy examples. Catches malformed candidates in seconds before they hit the expensive real eval.
5. The proposer itself: Claude Code (or any coding agent with real filesystem/tool access) driven by a short natural-language “skill” describing directory layout, what it can/can’t touch, and the objective(s) — but not how to diagnose failures. The paper is explicit that the quality of this skill text mattered more to search quality than iteration count or population size, and recommends running a few 3–5-iteration debug passes on the skill before committing to a full search run.
6. An orchestrator loop: propose k candidates → validate → evaluate → log → repeat for N iterations → take the Pareto frontier by search-set score → run one held-out test evaluation on the winner(s).

The genuinely hard parts are (a) building a search set that’s hard-but-cheap, and (b) writing a skill that constrains outputs (safety, forbidden files, the objective) without constraining how the agent diagnoses problems — over-specifying the second one is exactly the hand-designed-search-structure trap this paper is arguing against.

How to Improve It

1. Ablate the proposer. Swap Claude Code + Opus-4.6 for other coding agents (open-weight or commercial) to separate “the filesystem-access idea works” from “this specific agent is unusually good at it” — the paper flags this as open.
2. Add a real held-out split for benchmark-discovery settings. The TerminalBench-2 result searches and tests on the same 89 tasks; splitting even a small contested benchmark in half (accepting the cost) would let you report a cleaner generalization number alongside the current discovery number.
3. Cost-normalize against the text optimizers it beats. Report $/iteration or $/final-score against GEPA and OpenEvolve at matched compute budgets, not just matched evaluation counts, given the ~1000x gap in diagnostic tokens per step.
4. Build the “queryable CLI” the appendix mentions but doesn’t fully build out. As the archive grows past what plain grep/cat can navigate efficiently, a small index (Pareto frontier, top-k, diff-two-runs) would keep search-time cost from scaling with archive size — the paper calls this optional; at scale it likely isn’t.
5. Co-evolve harness and weights. The discussion explicitly proposes this as future work: alternate between fixing the harness and fine-tuning/RL-training the model on rollouts collected under it, then re-searching the harness against the updated model — closing the loop the paper currently leaves open (harness search only, frozen model).

Glossary

• Harness — the code wrapped around an LLM that decides what it’s shown, what gets stored, and when something is retrieved; the thing being optimized here (not the model weights).
• Coding agent — an LLM-based system that can call developer tools and edit code directly, as opposed to a raw LLM that just returns text from a prompt.
• Proposer — the coding agent (Claude Code + Opus-4.6 in this paper) responsible for reading prior experience and writing new harness candidates.
• Text optimizer — a general class of automated tools (OPRO, TextGrad, GEPA, AlphaEvolve/OpenEvolve, Feedback Descent, TTT-Discover) that iteratively improve a text or code artifact using feedback from previous attempts.
• Search set vs. test set — the search set is the subset of tasks used during the search loop to score candidates and drive proposals; the held-out test set is only touched once, at the very end, and the proposer never sees it.
• Pareto frontier — the set of candidates where no other candidate is better on every objective at once (e.g., accuracy and context cost); instead of picking one number to optimize, you keep the whole tradeoff curve.
• Interface validation — a fast smoke test (import, instantiate, run on a couple of examples) that filters out broken candidates before paying for a full evaluation.
• ACE (Agentic Context Engineering) — a hand-built baseline system that curates and reflects on memory over time; the strongest prior hand-designed harness compared against in text classification.
• MCE (Meta Context Engineering) — another hand-built baseline that maintains and evolves a library of natural-language “skills” for context construction.
• BM25 — a classic lexical (keyword-overlap-based) text retrieval scoring function; used as the underlying retrieval mechanism in the discovered math harness, as opposed to a neural embedding-based (“dense”) retriever.
• TF-IDF — a simpler lexical similarity measure (term frequency × inverse document frequency) used by one of the discovered text-classification harnesses to find similar past examples.
• Dense retrieval — retrieval using a neural embedding model to find semantically similar text, as opposed to BM25/TF-IDF’s keyword overlap.
• GEPA — a text optimizer that reflects on rollout traces for a single candidate at a time; the closest prior method in terms of feedback richness, but still far more limited than Meta-Harness’s full-archive access.
• AlphaEvolve / OpenEvolve — evolutionary code-search systems that mutate programs using LLMs guided by a program database and scalar scores; designed for stateless algorithm discovery rather than stateful harnesses.
• TerminalBench-2 — a benchmark of 89 long-horizon, autonomous command-line tasks used to evaluate agentic coding harnesses.
• Terminus-KIRA / Terminus 2 — strong hand-engineered agent harnesses for TerminalBench-2 that Meta-Harness initializes its search from and is compared against.
• Opus-4.6 / Haiku-4.5 — two Claude model sizes used as the frozen base models being wrapped by discovered harnesses in different experiments (Opus as the stronger, more expensive model; Haiku as the weaker, cheaper one).
• Non-Markovian (in this context) — the proposer’s decisions depend on a broad slice of search history, not just the single most recent candidate (“the parent”) — confirmed empirically by the fact it reads dozens of prior files per iteration rather than just the last one.