Self-Harness: Harnesses That Improve Themselves

TL;DR

An LLM agent is two things stacked together: the model (fixed weights) and the harness (the prompts, tools, memory, and runtime rules wrapped around it). Today, humans hand-build the harness, and a harness that works for one model is often wrong for another. This paper asks: can the same fixed model improve its own harness? Their system, Self-Harness, runs a loop — find recurring failure patterns from execution traces, propose a few small targeted harness edits, and promote only the edits that pass a held-out regression test. On Terminal-Bench-2.0, across three different model families, held-out pass rates jumped by large margins (e.g. MiniMax M2.5 from 40.5% to 61.9%, Qwen3.5 from 23.8% to 38.1%, GLM-5 from 42.9% to 57.1%). The edits were not generic “try harder” prompts; each model got different fixes matched to its own recurring mistakes.

Problem & Motivation

Here is the concrete pain. You run three different LLMs behind the same agent scaffold. One of them keeps forgetting to write the output file the grader checks for. Another gets stuck retrying the exact same broken shell command twenty times. A third loses its environment settings (a PATH change) between shell calls and then can’t find the tool it just installed. None of these are “the model is dumb” problems — they are harness problems. The model response is fine; the surrounding rules failed to catch a behavior, provide a recovery action, or nudge the agent at the right moment.

Historically, a human expert notices these patterns and hand-edits the scaffold: adds a system-prompt line, sets a retry limit, adds a verification step. That worked when there were a handful of models. It does not scale now: models ship weekly, each has its own quirks (tool-use habits, error modes, prompt sensitivities), and each one needs a different harness. Hand-tuning a bespoke harness per model is a treadmill.

Two escape hatches already exist, and both have a catch:

• Meta-Harness / observability-driven evolution [5, 8] — use a stronger external agent to optimize a weaker agent’s harness. Catch: for a frontier model there is no stronger agent to supervise it, and an external optimizer may not share the target model’s actual failure modes.
• Self-improving agents (Reflexion, agentic context engineering, STOP) [23, 34, 31] — the agent adapts its memory, context, or generated program over attempts. Catch: the thing being improved is a response strategy or a scratchpad, not the declared harness itself — the tools, policies, and runtime rules.

Self-Harness targets exactly that gap: let the target model, running under its own current harness, propose a bounded, testable edit to the harness that governs its own future behavior. No human. No smarter model. No weight updates.

What’s New (Core Contribution)

1. The paradigm itself: the agent edits its own harness (a “third way”).

   • Before: harness improvement was either human engineering (an expert edits the scaffold) or external optimization (a stronger agent searches over harness designs).
   • Now: the same fixed model is put in a “proposer” seat and asked to edit the very scaffold it runs inside. This is a genuinely new position on the map (Figure 1) — self-directed, not externally supervised.

2. Failure mining that separates “what the grader rejected” from “what the agent actually did wrong.”

   • Before: automated approaches tended to patch coarse outcomes (“timeout”, “missing file”) or feed the proposer raw logs.
   • Now: each failure gets a structured signature — (verifier cause, causal status of the agent behavior, reusable agent mechanism) — and failures are clustered by exact signature match. Two runs that both “timed out” are not grouped if the underlying behavior differs. This gives the proposer clean cross-case evidence about a reusable weakness, not one-off anecdotes.

3. A conservative acceptance rule that blocks silent regressions.

   • Before: self-improvement loops often accept an edit if total score goes up.
   • Now: an edit is promoted only if it helps at least one task split without hurting the other (formally: both split-deltas ≥ 0 and at least one > 0). An edit that robs the held-out set to pay the held-in set is rejected even if the total rises. Every promotion is logged and auditable — a harness lineage (h₀ → h₁ → h₂ …) of small reversible changes.

4. Evidence that different models need different harnesses — measured, not asserted.

   • The retained edits diverge sharply per model (early-artifact creation for MiniMax, dependency-precheck + retry-discipline for Qwen3.5, environment-persistence for GLM-5), which is direct evidence for the “harness design is model-specific” claim that motivated the whole thing.

How It Works (Technically)

The setup, in plain terms. You hold two things fixed: the model M (its weights never change) and the evaluator E (the grader that runs a task and returns pass/fail). The only thing allowed to change is the harness h. Because everything else is frozen, any change in pass rate can be blamed on the harness — that is the whole experimental point.

Run one task x through model M under harness h and you get a trace τ (every message, tool call, and verifier check) and an output y. The evaluator turns that into an outcome z = E(x, τ, y), e.g. pass or fail. Self-Harness walks a chain of harnesses h₀ → h₁ → h₂ …, where each arrow is one small, tested edit to the scaffold — never a weight update.

The loop has three stages. Let me walk each, then trace one real task end to end.

Stage 1 — Weakness Mining (turn failures into structured evidence). Run M under the current harness hₜ on the held-in split D_in. Collect the trace record for every task: rᵢ = (xᵢ, τᵢ, yᵢ, zᵢ). Keep only the failures: Fₜ = { rᵢ : zᵢ = fail }. Now the key move — don’t treat failures as isolated anecdotes. For each failed run, compute a failure signature:

$$\varphi(r_i) = (c_i,; q_i,; m_i)$$

Read it as three plain questions:

• cᵢ — what did the grader ultimately reject? (the terminal verifier-level cause, e.g. “required artifact missing”).
• qᵢ — was the agent’s behavior actually the cause? (the causal status of that behavior in the trace — did it lead to the rejection, or was it incidental?).
• mᵢ — which reusable behavior mechanism is this? (an abstract pattern like “kept exploring instead of writing output”, “retried the same failing command”).

Then cluster failures by exact agreement of the signature: C_φ = { rᵢ ∈ Fₜ : φ(rᵢ) = φ }. Because it’s exact-match on a structured triple, clustering is deterministic — no fuzzy embedding similarity. This is deliberate: two runs that share a symptom (“timeout”) but differ in mechanism land in different clusters, because they need different harness fixes. Each cluster becomes a failure pattern carrying its size, example tasks, shared symptoms, verifier evidence, and the inferred mechanism. Clusters are ranked by support (how many cases) and estimated actionability (how likely a small harness edit fixes it). The output is an evidence bundle Bₜ. Crucially, Bₜ describes problems, it does not prescribe fixes — that keeps the “diagnosis” role clean and separate from the “propose a fix” role.

Stage 2 — Harness Proposal (turn evidence into a few small candidate edits). Now the same fixed model M is invoked in a proposer role. It is not an unrestricted optimizer; it gets a bounded context: the editable surfaces of the current harness (the specific config points it’s allowed to touch), the failure patterns from Bₜ, a record of behaviors that are passing and must be preserved, and summaries of edits already tried. From this it generates K parallel, mutually distinct candidate bundles:

$$P_t = {(\Delta_j,, a_j)}_{j=1}^{K}, \qquad h_t^{(j)} = \Delta_j(h_t)$$

Each Δⱼ is an edit function — apply it to the current harness hₜ and you get candidate harness hₜ^(j). Each aⱼ is an audit record: which failure pattern it targets, which surface it edits, the expected behavioral effect, and the regression risks. Two constraints do the heavy lifting:

• Diversity across branches — the K candidates must target different mechanisms/surfaces/hypotheses, not reword the same idea. This broadens the search.
• Minimality within a branch — each edit touches only the surface needed for its one mechanism, preserves unrelated behavior, and avoids rewriting the control architecture. This keeps each candidate interpretable and cheap to test.
• A pattern is only a valid target if it is addressable — supported by evidence and plausibly fixable by editing a harness surface. Clusters that reflect raw task difficulty or model capability limits (no execution rule would help) are skipped, not forced into a patch.

Stage 3 — Proposal Validation (keep only edits that pass a regression gate). Each candidate harness is re-run under the same evaluator on both splits: D_in (held-in, the evidence the proposer saw) and D_ho (held-out, never shown to the proposer — a true regression test). Let Pin(h) and Pho(h) be the number of tasks passed on each split. Compute the deltas versus the current harness:

$$\Delta_{in}^{(j)} = P_{in}(h_t^{(j)}) - P_{in}(h_t), \qquad \Delta_{ho}^{(j)} = P_{ho}(h_t^{(j)}) - P_{ho}(h_t)$$

Accept the candidate only if:

$$\Delta_{in}^{(j)} \ge 0 ;\wedge; \Delta_{ho}^{(j)} \ge 0 ;\wedge; \max(\Delta_{in}^{(j)}, \Delta_{ho}^{(j)}) > 0$$

In English: don’t hurt either split, and help at least one. This is a conservative gate — a candidate that gains 5 on held-in but loses 3 on held-out is rejected, even though its total went up. That single rule is what stops the loop from overfitting to the failures it was shown. If evaluation is noisy, they repeat it and aggregate pass counts before applying the rule, so one lucky run can’t sneak an edit through. Candidates that don’t modify any editable surface, or crash before producing a result, are also rejected. Accepted edits in a round are merged into hₜ₊₁; rejected ones are logged but leave the active harness untouched. Then the loop repeats.

One concrete trace, start to finish (MiniMax M2.5, count-dataset-tokens task — Figure 7).

• Under the initial harness: the agent is asked to count tokens in a dataset. It finds the relevant metadata config… and then just keeps exploring the dataset. It never writes the required answer file. It runs out of budget and times out → fail.
• Weakness Mining: this failure (and others like it) gets signature ≈ (cause = required artifact missing, status = caused by agent, mechanism = kept exploring instead of producing output). Several tasks cluster here → a strong “missing artifacts” pattern in Bₜ.
• Harness Proposal: the proposer, seeing that pattern, generates a candidate edit Δ: change the bootstrap instruction from “identify the smallest edit surface” to “identify the required output artifact and create an initial version as early as possible”, and enable the runtime policy with a cap on total tool messages (redirect instead of exploring forever).
• Proposal Validation: re-run on both splits. Held-in and held-out pass counts both go up, neither drops → accept, merge into the next harness.
• Under the edited harness (same task, same model): the agent now identifies the metadata-backed subset, computes the token total, writes /app/answer.txt early, reads it back to verify, and stops. Pass.

Nothing about the model changed. A ~two-line change to its scaffold flipped a timeout into a pass — and, because the change was validated on held-out tasks, it generalized rather than memorizing this one task.

Architecture & data flow

flowchart TD
  H[Current harness h_t<br/>prompts / tools / memory / policies] --> RUN[Run fixed model M on held-in tasks]
  RUN --> TR[Execution traces + verifier outcomes]
  TR --> SIG["Failure signature phi = (verifier cause, causal status, agent mechanism)"]
  SIG --> CL[Cluster failures by exact signature match]
  CL --> B[Evidence bundle B_t<br/>ranked failure patterns, no fixes prescribed]
  B --> PROP[Same model M as PROPOSER<br/>bounded context: editable surfaces + patterns + passing behaviors]
  PROP --> K["K parallel candidates: diverse across, minimal within<br/>each = edit function delta_j + audit record a_j"]
  K --> VAL{Regression gate<br/>held-in delta >= 0 AND held-out delta >= 0 AND one > 0?}
  VAL -->|reject| LOG[Log candidate, harness unchanged]
  VAL -->|accept| MERGE[Merge accepted edits]
  MERGE --> H2[Next harness h_t+1]
  LOG --> H2
  H2 -.next iteration.-> RUN

The parts, at a glance

flowchart LR
  subgraph FIXED[Held FIXED all run]
    M[Model M<br/>weights frozen]
    E[Evaluator E<br/>deterministic verifier]
  end
  subgraph MUT[The ONLY thing that changes]
    HAR[Harness surfaces:<br/>system prompt, bootstrap,<br/>tools, verification rule,<br/>runtime policy, memory]
  end
  M --> AG[Agent run]
  HAR --> AG
  AG --> E
  E --> OUT[pass / fail + trace]
  OUT --> HAR

The Self-Harness loop as a running cycle: an evidence bundle feeds K parallel candidate edits, each is gated on held-in AND held-out deltas, and only accepted edits merge into the next harness. Watch candidates fail the gate (red) or pass (green); the harness version ticks up only on an accept. Schematic, not the paper's exact numbers.

The algorithm, simplified

# Self-Harness: a fixed model edits its own harness, one validated step at a time.
# llm_propose(...) -> list of edit functions;  evaluate(...) -> passes on each split.

def self_harness(M, h0, D_in, D_ho, E, K=4, rounds=15):
    h = h0
    for t in range(rounds):
        # --- Stage 1: Weakness Mining ---------------------------------------
        records = [run_and_grade(M, h, x, E) for x in D_in]      # trace + pass/fail
        fails   = [r for r in records if r.outcome == "fail"]
        # signature = (verifier cause, was-agent-the-cause, reusable mechanism)
        clusters = group_by(fails, key=lambda r: signature(r))   # EXACT-match clustering
        evidence = rank_patterns(clusters)                       # by support x actionability
                                                                 # (describes problems, not fixes)

        # --- Stage 2: Harness Proposal (same model M, proposer role) --------
        # diverse ACROSS branches, minimal WITHIN each branch
        candidates = llm_propose(M, editable_surfaces(h), evidence,
                                 preserve=passing_behaviors(records), k=K)

        # --- Stage 3: Proposal Validation (regression gate) ----------------
        accepted = []
        base_in, base_ho = evaluate(M, h, D_in, E), evaluate(M, h, D_ho, E)
        for edit in candidates:
            h_cand = edit(h)                                     # apply the bounded edit
            d_in = evaluate(M, h_cand, D_in, E) - base_in        # held-in delta
            d_ho = evaluate(M, h_cand, D_ho, E) - base_ho        # held-out delta (proposer never saw D_ho)
            # keep only if it helps one split and hurts NEITHER
            if d_in >= 0 and d_ho >= 0 and max(d_in, d_ho) > 0:
                accepted.append(edit)                            # else: log + discard

        h = merge(h, accepted) if accepted else h                # lineage grows only on accepts
    return h

The whole contribution lives in three lines: signature(r) (diagnose the mechanism, not the symptom), the d_in >= 0 and d_ho >= 0 and max(...) > 0 gate (promote without regressing), and the fact that llm_propose is the same M editing its own scaffold.

Built on Prior Work

Results & Evidence

Setup. Terminal-Bench-2.0, a 64-task subset (dropped multimodal + flaky-network tasks) of containerized terminal jobs graded by deterministic verifiers over the final container state. Three models from different families: MiniMax M2.5, Qwen3.5-35B-A3B, GLM-5. Same minimal DeepAgent harness as the start point for all three. Metric: Pass (%) over two repeated attempts. Every comparison is within-model (only the harness varies).

Headline numbers (held-out split — the honest one, never shown to the proposer):

What’s genuinely convincing:

• All three models improve on held-out, and no promoted harness degrades either split — exactly what the acceptance rule was designed to guarantee, so the design and the result are consistent.
• The edits are model-specific and legible (Figures 5, 6, 10). MiniMax got “create output early” + a tool-call cap; Qwen3.5 got dependency-precheck, retry-discipline, and a tool-error-triggered middleware guard; GLM-5 got environment-persistence + an “exploration → implementation” nudge. Different pathologies, different fixes — from the same starting harness.
• Before/after traces (Figures 7, 8, 9) show the behavior actually changing in the diagnosed way, not just a score moving.
• It reaches the final harness in a handful of gated edits, not a long lucky streak — the trajectory plots show many rejected candidates and a few accepted ones.

What the evidence does NOT establish (read this before you sell it):

• One benchmark, one task type. Only Terminal-Bench-2.0 (terminal/CLI tasks). No web, GUI, multi-agent, or long-horizon coding evidence. The authors say so plainly.
• Small absolute set. 64 tasks split into held-in/held-out; pass counts are small integers, so a “+1 pass” moves the percentage a lot. Two repeats is thin for a stochastic system.
• The gate is only pass-rate non-regression. It does not check safety, cost, latency, or side effects. The authors flag that “higher-stakes harness changes would require stronger acceptance gates.”
• No comparison to human-tuned or Meta-Harness baselines on these same tasks — so we know Self-Harness beats its own minimal start, not that it matches an expert.
• Held-out is held out from the proposer, but not from the loop — it’s used by the promotion gate every round, so across many rounds the process can still drift toward benchmark-specific patterns (the authors acknowledge this).

Net: a clean, well-controlled proof-of-concept that self-editing harnesses work and generalize within a domain. Not yet evidence of open-ended or cross-domain self-improvement.

How You’d Use It

For an AI-services shop, this is a maintenance-cost lever and a small moat, more than a headline product.

• Model-swap insurance. Every time you change the model behind a client agent (cheaper model, new release, on-prem swap), the old hand-tuned prompt/policy stack is silently mis-fit. A Self-Harness loop re-tunes the scaffold to the new model automatically, gated so it can’t regress. That turns a scary re-engineering job into an overnight batch run.
• A “hardening” service. Offer clients: “give us your agent + a graded task set, we’ll evolve a model-specific, regression-tested harness and hand you an audit log of every change and why.” The lineage of small, reversible, evidence-backed edits is exactly what a risk-averse enterprise buyer wants to see.
• Turn your eval set into an asset. The loop is only as good as your verifier and your held-out split. If you already build deterministic evals for clients (you should), Self-Harness is the thing that consumes them to produce value — it monetizes eval work you’re already doing.
• Where it slots in your MAS. In a multi-agent system, run one Self-Harness loop per role (planner, coder, verifier), each with its own model and its own graded sub-tasks. The signature-clustering step is exactly the “why do our agents keep failing the same way” analysis you probably do by hand today — now automated into concrete config edits.

Honest scoping for a pitch: this improves reliability on tasks you can grade automatically. It does not invent new capabilities, and it needs a real verifier. Sell it as “we make your existing agents fail less, provably, and we keep the receipts.”

Build Your Own (Minimal Recipe)

You can stand up an 80% version in a week if you already have graded tasks. Components, in build order:

1. A gradeable task set + a deterministic verifier. This is the hard prerequisite. Split it into held-in (proposer sees the failures) and held-out (only the gate sees it). Without an automatic pass/fail signal, none of this works — start here.
2. A harness with explicitly editable surfaces. Don’t let the proposer rewrite arbitrary code. Expose named knobs: system_prompt, bootstrap_instruction, verification_instruction, failure_recovery_instruction, runtime_policy (e.g. max_total_tool_messages), tools, memory_sources. The paper’s Figure 3 is literally a set of build_*() functions returning strings/dicts — copy that shape.
3. A trace logger. Capture messages, tool calls, and the verifier’s terminal reason per run. You need the reason, not just pass/fail.
4. A weakness-miner. For each failure, prompt the model to emit the signature triple (verifier_cause, was_agent_the_cause, reusable_mechanism) as structured JSON, then group by exact triple. Rank clusters by size. (Exact-match grouping is a feature — it keeps clustering deterministic and cheap. You can start even simpler: cluster on verifier_cause alone, add the mechanism field once that’s too coarse.)
5. A proposer prompt. Feed it: the editable-surface schema, the top failure clusters, a note of what’s currently passing (don’t break it), and past attempts. Ask for K distinct edits, each as a JSON patch to one surface plus an audit record. Sampling K in parallel at moderate temperature gives you the diversity.
6. A validation harness. Apply each patch, re-run both splits, compute the two deltas, apply the gate d_in >= 0 and d_ho >= 0 and max(...) > 0. Merge accepted patches; log everything.

The two genuinely hard parts:

• The verifier. Everything rests on a trustworthy, deterministic pass/fail. Flaky graders will promote garbage. Budget most of your effort here.
• Getting minimal-but-real edits. Left alone, the proposer writes vague “be more careful” prose that passes the gate by luck and doesn’t generalize. Enforce: one surface per edit, must name the target mechanism, must be a concrete config change (a policy value, a specific instruction), not a mood.

Reach for: any agent SDK with declarative config (LangChain DeepAgents is what they used; or roll your own dict-of-surfaces), a container runner for isolated task execution, and structured-output prompting (JSON mode) for the miner and proposer.

How to Improve It

1. Add cost/latency/safety to the gate (highest value). The current rule only checks pass-rate non-regression. Make acceptance multi-objective: d_pass >= 0 AND tokens_delta <= budget AND no_new_destructive_actions. This is the single change that moves it from “demo” to “deployable,” and it’s directly testable — just extend the acceptance predicate.
2. Cross-model transfer of edits. They prove edits are model-specific, but some (“create output early”) are clearly general. Test: take edits accepted for model A, apply to model B as seed candidates, and measure how many survive B’s gate. If many do, you get a reusable “harness edit library” and warm-start every new model.
3. Beat overfitting-to-the-benchmark with a rotating held-out. Because held-out is used by the gate every round, it slowly leaks. Rotate the held-out split each round (k-fold over your task set), or keep a third, truly-untouched split you only measure at the very end. Cheap to add, tightens the generalization claim.
4. Let edits compose and prune. The merge step just unions accepted edits; over many rounds the harness bloats and edits may interact. Add a periodic ablation pass — drop each retained edit and re-test; remove any whose removal doesn’t hurt. Keeps the harness minimal and interpretable.
5. Diagnose the cases the gate keeps rejecting. Rejected clusters that are “non-addressable” (task difficulty / model capability limits) are currently just skipped. Route those to a different lever — a tool addition, a sub-agent, or “this needs a stronger model” — turning the reject pile into a capability-gap report you can hand a client.

Glossary

• Harness — everything wrapped around the model to make it an agent: system prompt, bootstrap/verification/recovery instructions, tools, memory, runtime policies. Not the model weights.
• Base model / fixed model (M) — the LLM whose weights never change during the whole process; only its scaffold changes.
• Evaluator / verifier (E) — the deterministic grader that runs a task’s final state and returns pass/fail. Held fixed.
• Execution trace (τ) — the full log of one agent run: messages, tool calls, and verifier checks.
• Failure signature (φ) — a structured triple (verifier cause, causal status, reusable mechanism) used to group failures by why they failed, not just that they failed.
• Evidence bundle (Bₜ) — the ranked set of failure patterns handed to the proposer; describes problems, prescribes no fixes.
• Editable surface — a named, allowed-to-change part of the harness (e.g. bootstrap_instruction, runtime_policy). Edits are confined to these.
• Edit (Δⱼ) — a function that maps the current harness to a candidate harness; a bounded patch to one surface.
• Audit record (aⱼ) — the metadata for an edit: which failure it targets, what surface it touches, expected effect, regression risk.
• Held-in / held-out split — held-in tasks supply failures to the proposer; held-out tasks are hidden from the proposer and used only by the promotion gate as a regression test.
• Acceptance / promotion rule — promote an edit only if it helps at least one split and hurts neither (d_in ≥ 0, d_ho ≥ 0, max > 0).
• Regression test — re-running on tasks the edit wasn’t designed for, to catch edits that fix one thing and break another.
• Harness lineage — the chain h₀ → h₁ → h₂ … of small, logged, reversible harness versions.
• Meta-Harness — prior approach where a stronger external agent optimizes a weaker agent’s harness; Self-Harness removes the external agent.
• DeepAgent — the LangChain agent SDK used as the minimal starting harness (Figure 3).
• Terminal-Bench-2.0 — benchmark of containerized command-line tasks graded by deterministic verifiers, used as the test-bed.