Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents

TL;DR

An LLM agent runs a real task once, fails, and the failure is thrown away. That is the waste this paper attacks. Evo-Harness keeps the model frozen and instead grows an external file of skills — short written rules with a “when to use me” trigger. After each batch of tasks it reflects only on the failures, proposes candidate lessons, and a second model (the “evolver”) decides whether to add, merge, revise, or drop each one into a hard-capped library. On five hard benchmarks (TerminalBench-2, SWE-bench Lite, CL-Bench, τ-bench, WebArena-Infinity) this beats both a no-learning baseline and five prior experience-reuse methods, with the biggest jump on command-line work (62.9% → 73.0%).

The single most useful finding is not the win. It is the failure mode they isolated: when the agent judges its own success instead of reading a real verifier, learning makes it worse than not learning at all (29.5% → 28.0%). Self-graded self-improvement is a trap. Grounded feedback is the whole ballgame.

Problem & Motivation

The pain in one sentence: in production, an agent sees each task once, and a failed run produces a huge, messy transcript that nobody converts into anything the agent can use next time.

Break that into its three separate difficulties, because prior work usually only solves one:

1. One-shot. Real task streams do not let you retry the same ticket fifty times to find a good policy. Reflexion-style loops assume you can retry until you pass. In a live stream, the task is gone the moment it is closed. You get exactly one execution context to learn from.
2. Noisy context. One SWE-bench execution is a 40-step transcript: file reads, greps, three wrong patches, stack traces, tool timeouts, the model narrating its own confidence. Buried in that is maybe one transferable lesson (“read the failing test’s exact assertion before editing”). Everything else — the repo name, the file path, the specific stack trace — is anti-knowledge: writing it down makes the agent worse on the next task, because it overfits to a repo it will never see again.
3. Nobody knows what actually drives the gains. Papers in this space report “our memory helps, +3 points.” They rarely separate whether the win came from the retrieval, the writing, the model, or just from having more tokens in the prompt.

Point 3 is the one the authors care most about, and it is the honest justification for another skill-library paper. Their own results make the case brutally: in Table 1, four of six prior methods score below the no-learning baseline on at least one benchmark. AWM drops τ-bench from 72.7 to 70.9. ACE drops TerminalBench from 62.9 to 61.8. So “let the agent learn from experience” is not a free win — badly compiled experience is an active poison in the context window. That is the gap.

What’s New (Core Contribution)

Four claims. I rate them honestly below.

1. “Online harness learning” as a formal setting. Before: collect a pile of trajectories, mine them offline, ship a static skill library. Now: tasks arrive as a stream of batches B₁…B_K; before each batch the agent has harness H_i; after the batch it becomes H_{i+1}; the model weights never change. — Modest novelty. This is continual prompt-level learning, which people have been doing informally (every hand-maintained CLAUDE.md is this). Writing it down as a setting with a fixed protocol is genuinely useful for making comparisons fair, and it forces the one-shot constraint that most prior work quietly dodges.

2. Context-to-harness skill compilation with two levels. Before: one flat store of memories/cases you retrieve from. Now: candidate lessons get compiled into general skills (cross-task patterns, found by comparing memories across tasks in the batch) and topic skills (localized procedures for a recurring task type/interface/domain). Crucially the level is not a label the reflector assigns — it emerges from the evolver noticing a pattern in more than one context. — This is the real mechanical contribution. It is a promotion rule: local until proven general.

3. Hard budgets that force merging. Max 5 general skills, max 5 skills per topic, batch of 16. — Buried in Appendix F and barely discussed, but I think this is the actual engine. A cap means the evolver cannot hoard; every new lesson must beat an incumbent or be merged into it. Compression is doing the noise filtering, not the prompt wording.

4. A factor-isolation study. They ablate the evolver design, the feedback source, the solver/evolver pairing, and train→test transfer. — The most valuable part of the paper, and the part that would survive if the method itself were forgotten.

What is repackaged, plainly: reflect-on-failure is Reflexion (2023). A curated skill library is Voyager (2023). An evolving guidance file injected into the prompt is Dynamic Cheatsheet (2025). {ADD, MERGE, REVISE, SKIP} is the standard memory-management operator set. There is no new loss, no training, no RL, no new architecture — this is a prompt pipeline with two LLM roles and a size cap. The authors are reasonably upfront about this; their pitch is the setting plus the analysis, not a new algorithm. Take the paper as a well-run experiment, not an invention.

How It Works (Technically)

There is no gradient anywhere in this paper. Every “learned” object is text. Keep that in mind and the math gets easy.

The state and the loop

The learned object is the harness — a set of guidance entries:

$$H_i = {h_i^1, \dots, h_i^{n_i}}$$

Plain English: H_i is just a folder of markdown files at time i. Appendix F says exactly that — markdown skill files with light YAML metadata. Each entry has four fields:

Now the per-task loop. Five equations, each one line of code in practice:

(1) Select. S_ij = Select(x_ij, H_i; b), with |S_ij| ≤ b. Retrieve at most b harness entries whose trigger matches this task. Implementation detail from the appendix: they use Claude Sonnet 4.5 as the selector in every experiment, even when the solver is Opus. So selection is an LLM read of the trigger lines, not embedding cosine similarity. b is the injection budget — the token allowance for guidance.

(2) Inject. x̃_ij = Inject(x_ij, S_ij). Paste the selected skills into the task prompt. That is it. This is the entire mechanism by which learning reaches the agent.

(3) Execute. (τ, y, f) = A(x̃_ij). The frozen solver A runs. τ is the action trajectory, y is the outcome the agent claims, f is external feedback — unit test results, a Docker verifier, a state checker, a rubric judge. Note y and f are separate variables and that separation is the paper’s sharpest idea: the agent’s claim about what it did is not the same object as the environment’s judgment.

(4) Context. c_ij = (x_ij, τ_ij, y_ij, f_ij). The full noisy record. This is the raw material.

(5) Reflect — failures only. r_ij = Reflect(c_ij). Fires only when the execution failed or got negative feedback. Successes are deliberately skipped, on the argument that a success transcript is mostly task-specific detail, while a failure marks the exact boundary of what the solver currently cannot do. Output is a candidate memory:

$$r = (\texttt{lesson}, \texttt{trigger}, \texttt{evidence}, \texttt{scope_hint})$$

Critically, reflection does not see the current harness. It cannot tell you whether the lesson is new. It only produces a failure-grounded candidate. That job is deliberately pushed to the next stage.

The evolver — where compilation actually happens

At the end of a batch you have R_i = {r_i1 … r_im} — up to 16 candidate lessons, many redundant, several wrong. The evolver takes both:

$$O_i = \text{Evolver}(H_i, R_i), \qquad o = \pi_{\text{evolve}}(r, H_i, R_i) \in {\textsc{Add}, \textsc{Merge}, \textsc{Revise}, \textsc{Skip}}$$

π_evolve looks like a policy from reinforcement learning, but it is not trained — no reward, no gradient, no value function. It is a prompt that asks a model to pick one of four verbs. Reading it as a policy is notation borrowed for flavor; treat the symbol as “the LLM’s decision rule.”

The important part is what each argument gives it:

• Conditioning on H_i (the current harness) enables MERGE and REVISE — the evolver can see it already has a rule about this and strengthen it instead of duplicating.
• Conditioning on R_i (the whole batch of candidates at once) enables the promotion decision. One task failing on unverified state is an anecdote; six tasks in the same batch failing on unverified state is a cross-task pattern worth a general skill.

Then two compile passes run over the same candidates (Algorithm 1, lines 12–13):

• CompileTaskType(H_i, R_i) → topic skills. Localized: “in this app, disabling an auto-label is a different control from deleting it.”
• CompileCrossTask(H_i, R_i) → general skills. Only fires when the same shape of failure shows up in multiple contexts. The appendix prompt is explicit: “Create or update a general skill only when a pattern appears across multiple contexts… must avoid context-specific references.”

Finally H_{i+1} = ApplyEdits(H_i, O_i), and the budget bites: 5 general skills total, 5 per topic. When those are full, ADD is impossible; the evolver must MERGE or SKIP. The cap is the noise filter. A rule only survives if it keeps earning its slot against competitors.

Architecture

flowchart LR
  T[Task batch B_i<br/>16 tasks] --> SEL[Select<br/>Sonnet 4.5<br/>trigger match, budget b]
  H[(Skill Harness H_i<br/>markdown + YAML<br/>5 general / 5 per topic)] --> SEL
  SEL -->|injected skills| SOLV[Frozen Solver A<br/>Opus 4.6]
  SOLV --> CTX[Execution context c<br/>input, trajectory, outcome, feedback]
  ENV[Environment verifier<br/>unit tests / Docker / state check] -->|feedback f| CTX
  CTX -->|failures only| REF[Reflect<br/>lesson, trigger, evidence, scope]
  REF -->|candidates R_i| EVO[Evolver<br/>ADD / MERGE / REVISE / SKIP]
  H --> EVO
  EVO -->|edits O_i| H

Data flow: one noisy trajectory, end to end

Here is the compilation traced on a real case from the paper’s appendix — WebArena task h10.

flowchart TD
  A["Task h10: create auto-label 'Legal',<br/>disable auto-label 'Support Ticket'"] --> B["Select from H_i<br/>early harness = nothing matches"]
  B --> C["Solver runs ~40 browser steps<br/>creates 'Legal', clicks Delete on 'Support Ticket'"]
  C --> D["Claimed outcome y:<br/>'both subgoals complete'"]
  C --> E["Verifier feedback f:<br/>FAIL - 'Support Ticket' still enabled"]
  D --> F["Context c = noisy 40-step transcript<br/>+ a false success claim"]
  E --> F
  F --> G["Reflect fires (failure)<br/>discard: DOM ids, URLs, step count, label names<br/>keep: create != disable; never trust own claim"]
  G --> H["Candidate memory r<br/>trigger: 'task changes a settings toggle'<br/>lesson: create and disable are distinct ops;<br/>re-open the settings list and confirm final state<br/>evidence: h10 verifier diff | scope_hint: topic"]
  H --> I{"Evolver sees r<br/>+ 15 sibling candidates"}
  I -->|"topic shelf full (5/5)"| J["MERGE into existing<br/>'auto-label operations' topic skill"]
  I -->|"same 'claimed done, never checked' shape<br/>also in tau-bench + terminal candidates"| K["ADD general skill:<br/>'verify terminal state independently<br/>of the agent's own narration'"]
  J --> L[(Harness H_i+1)]
  K --> L
  L --> M["Next batch: both skills match the trigger,<br/>get injected, solver re-checks state -> verifier passes"]

Read the discard line in step G again — that is the whole paper. Of a 40-step transcript, roughly one sentence survives compilation. The label names, the app, the click coordinates and the step count are all thrown away on purpose, because keeping them is what made prior methods score below baseline.

One batch, as message passing

sequenceDiagram
  participant H as Harness (files)
  participant S as Selector (Sonnet 4.5)
  participant A as Solver (frozen Opus)
  participant E as Env verifier
  participant V as Evolver
  loop each of 16 tasks in batch B_i
    S->>H: match task against triggers
    H-->>S: <= b skills
    S->>A: task + injected skills
    A->>E: actions
    E-->>A: pass / fail + diagnostics
    A->>V: context (only if failed) -> candidate memory
  end
  V->>H: read current harness
  V->>V: CompileTaskType + CompileCrossTask over all 16 candidates
  V->>H: ADD / MERGE / REVISE / SKIP under the 5-slot cap

Schematic of one batch through the compilation funnel: 16 executions → only failures reflect → candidate lessons → evolver verdicts → a harness that stays capped. Watch the ADD arrow dry up as the shelves fill; from then on every lesson has to merge into an incumbent. That squeeze is what filters noise. Counts are illustrative, not the paper's logged numbers; the batch size (16) and the 5-slot caps are the paper's.

The paper's Table 4, drawn to scale. The dashed line is "don't learn at all." Self-graded feedback lands below it on both benchmarks — the only setting in the paper where self-improvement is actively harmful. Hover a bar for the number.

The two-level harness as a structure you can orbit. Top plane: up to 5 general skills, promoted only when a failure shape recurs across topics. Lower plane: topic shelves (one per task type/interface), each capped at 5. Grey lines are promotions — a lesson starts local and moves up only if the evolver sees it in more than one context. Schematic layout, not the paper's actual inventory.

The algorithm, simplified

# Evo-Harness: the whole method. No training, no gradients — text in, text out.
# Contracts: llm(prompt) -> str ; solver(prompt) -> (trajectory, claim, feedback)
# harness.general: list[Skill] (cap 5) ; harness.topics: dict[str, list[Skill]] (cap 5 each)

BATCH, GEN_CAP, TOPIC_CAP, INJECT_BUDGET = 16, 5, 5, 3

def run_stream(tasks, harness, solver):
    for batch in chunks(tasks, BATCH):
        candidates = []
        for task in batch:
            skills = select(task, harness, k=INJECT_BUDGET)   # LLM matches task -> triggers
            traj, claim, feedback = solver(inject(task, skills))

            # ONLY failures teach. A success transcript is mostly task-specific noise.
            # `feedback` is the ENVIRONMENT's verdict (tests/verifier), never the agent's `claim`.
            if feedback.failed:
                candidates.append(reflect(task, traj, claim, feedback))

        evolve(harness, candidates)          # batch-level: needed to spot cross-task patterns
    return harness

def reflect(task, traj, claim, feedback):
    # Deliberately does NOT see the harness -> it cannot judge novelty, only extract a lesson.
    raw = llm(f"""This run failed. Verifier said: {feedback.details}
                  Trajectory: {compress(traj)}   Agent claimed: {claim}
                  Name the ONE missing action or wrong assumption.
                  Reject: generic advice, basic tool usage, exact replay of this task.
                  Return: lesson, trigger, evidence, scope_hint(general|topic).""")
    return parse(raw)

def evolve(harness, candidates):
    # Sees ALL candidates at once — that is what makes promotion possible.
    for topic, group in group_by_topic(candidates):
        shelf = harness.topics.setdefault(topic, [])
        for c in group:
            verdict = llm(f"""Existing skills: {shelf}  New proposal: {c}
                              Budget {TOPIC_CAP} — shelf currently {len(shelf)}/{TOPIC_CAP}.
                              ADD / MERGE / REVISE / SKIP? Prefer MERGE over duplication.""")
            apply_edit(shelf, c, verdict, cap=TOPIC_CAP)   # full shelf => ADD is unavailable

    # Promotion: a lesson becomes GENERAL only if its failure shape recurs across topics.
    shapes = cluster_by_failure_shape(candidates)
    for shape in shapes:
        if shape.spans_multiple_topics():
            general = llm(f"Write this as one context-free rule, no app/repo/file names: {shape}")
            apply_edit(harness.general, general, verdict="ADD_OR_MERGE", cap=GEN_CAP)

The two lines that carry the paper: if feedback.failed (grounded, external, failure-only) and cap=TOPIC_CAP (bounded, so growth forces compression). Everything else is plumbing you have already written.

Built on Prior Work

Lineage in one line: Reflexion’s critique + Voyager’s library + Dynamic Cheatsheet’s injected file, minus the retry, plus a size cap and a controlled experiment.

Results & Evidence

Headline (Table 1, Claude Opus 4.6 solver, success rate %):

Consistent, never negative, biggest where tasks have reusable procedure (shell work, tool calls). Averaged over three runs.

The four findings that matter more than the table:

1. Grounded feedback is mandatory (Table 4). Self-generated judgment: CL-Bench 29.54 → 27.96, SWE-bench 63.67 → 61.67. Both below not learning. Minimal (bare pass/fail) and Standard (error traces, failing assertions) both help. Interesting wrinkle: on SWE-bench, bare pass/fail (67.33) narrowly beat rich diagnostics (67.00) — detailed stack traces can glue a lesson to one repo. On CL-Bench rich feedback won by 4 points. So richer is not automatically better; it depends on whether the diagnostics are transferable.

2. The two levels do different jobs (Table 3). On heterogeneous reasoning (CL-Bench), Topic-Only reaches 33.70 vs. General-Only 30.28. On repo debugging (SWE-bench), General-Only 66.67 vs. Topic-Only 64.33. Neither shelf alone matches the full method. Practical read: if your own work spans many small varied domains, topic skills carry you; if it’s one deep codebase, general procedures do.

3. Skills transfer across models and splits (Fig. 5, SWE-bench). Skills evolved by Sonnet 4.5 on a training split, then used by an Opus 4.7 solver on unseen tasks: 68.8 → 73.4. Online updating with the same models: 75.0. So most of the value is transferable — you can evolve a harness cheaply with a small model and ship it to a big one. Live updating adds ~1.6 more points.

4. A weak solver cannot use guidance (Fig. 6). With an Opus solver, evolved skills help (70.7 → 75.3/76.0), and skills written by Sonnet helped Opus slightly more than Opus’s own (76.0 vs 75.3). With a Sonnet solver, every evolved harness hurt — 58.0 no-evolve vs. 55.3/55.7 with skills. The artifact is not the whole story; the solver has to be strong enough to contextualize a rule instead of following it off a cliff. If you are building on a cheap model, this technique may cost you points.

What the evidence does NOT establish — be skeptical here:

• No token-matched control. Injected skills add tokens to the prompt. There is no “inject an equal number of tokens of irrelevant or generic advice” baseline. Some of the gain could be prompt length, priming toward carefulness, or “verify your work” boilerplate rather than the specific compiled content.
• No single-skill causality. The authors say so themselves: several skills are injected together, and some runs do not even log which. The fail-to-pass percentages in Figure 7 (“this guidance appears in 100% of improved runs”) are correlational, and they label them as such. Do not read them as attribution.
• Ordering and seed sensitivity is untested. An online method’s harness depends on task order. One shuffle seed (42), three runs. Nobody re-shuffled the stream to see whether a different batch order produces a different harness and a different score. For a paper about online learning this is a real hole.
• Baselines may be under-tuned. Six prior methods, five of which score at or below no-evolve on multiple benchmarks. Either the field is genuinely that fragile, or the reimplementations are unflattering. No way to tell from the paper.
• A numeric inconsistency. SWE-bench Lite with an Opus 4.7 solver reads 68.8 (no-evolve) in Fig. 5 and 70.7 in Fig. 6, and 75.0 vs 75.3 for the evolved setting. Small, probably different subsets or run counts, but unexplained.
• Gains shrink with weaker models. Opus gets +3.7 to +4.5 on CL-Bench; Kimi-K2.5 gets +1.1, GPT-OSS +0.8. Combined with finding 4, the honest summary is: this technique amplifies strong models rather than rescuing weak ones.
• Cost is enormous and unreported per-run. ~$100K in API spend across the suite, 2–8 hours per benchmark configuration. There is no accounting of the marginal inference cost of the method itself — you are paying for a selector call, a reflection call, and an evolver call on top of every task.
• Scope limits the authors state: single-agent only, text-only, natural-language skills only. No multi-agent, no embodied, no executable skills.

How You’d Use It

You already run the substrate this paper describes. A skills directory with trigger lines, a selector that reads triggers, agents that inherit files — that is the harness. What this paper adds is a disciplined way to write into it automatically, and hard evidence about when doing so backfires.

Where it slots into a real harness:

The three concrete plays:

1. Your harness — self-writing skill files. You probably can’t fully describe your own agents’ operational knowledge, but their failures encode it. Run your agent on 50–200 real tasks with a real verifier, compile the failures, and let the pipeline write back a capped, human-readable, auditable skill file that plugs into your harness. Every rule cites the failure that produced it — that provenance line is what makes it defensible when you’re debugging why the agent changed behavior.
2. Cheap evolver, expensive solver. Finding 3 is the economic one. A Sonnet-class model can evolve a harness on a training split, and an Opus-class solver uses it profitably on unseen tasks. So the learning is cheap and offline; only the serving needs the expensive model. That means you can pre-bake a domain-specific harness once — for one codebase, one internal tool, one recurring workflow — and reuse it every time that model runs the task.
3. A compounding asset per domain. Each domain’s harness is small (5 general + 5-per-topic), portable text, and gets better with use. It’s the closest thing to a moat this technique offers for your own stack — not the pipeline, which is easy to rebuild, but the accumulated compiled failures from a domain you’ve run a thousand tasks in.

The disqualifier — check this before you build it. The method needs a programmatic verifier. Every benchmark here has one: unit tests, a Docker script, a state checker, a rubric judge. Table 4 shows that if you substitute the LLM’s own opinion of success, results go negative. So the qualifying question for any workflow you’d apply this to is: “can I mechanically tell whether this task succeeded?” If the answer is “a human reviews it eventually,” you either build the verifier first (that’s the real project) or you skip this technique. Running self-improvement without a verifier means running a system that quietly degrades.

Second disqualifier: if you’re set on a cheap open-weight solver, Figure 6 says the harness may cost you points. Know that going in.

Realistic effort to stand it up: the pipeline is small — reflect, evolve, select, inject is a few hundred lines, call it 3–5 days for a working version against an agent you already run. The verifier and the task stream are the actual work: 2–6 weeks depending on whether your success criteria are already machine-checkable. Then a 2–4 week shadow run before you let the harness write into production prompts. Budget the model spend honestly: three extra LLM calls per task, and the evolver call sees the whole batch, so it’s a long-context call.

Build Your Own (Minimal Recipe)

Smallest thing that captures ~80% of the value. You can have this working in a couple of days on top of an agent you already run.

Components, in build order:

1. A verifier and a task log. Do this first, and do not skip it. For each run, persist (task, trajectory, agent_claim, verifier_result, verifier_details). If you cannot produce verifier_result mechanically, stop — the rest is counterproductive (Table 4).
2. The harness as a directory. harness/general/*.md (cap 5) and harness/topics/<topic>/*.md (cap 5 each). YAML front matter: trigger, scope, evidence. Plain files, in git. Git gives you the diff-per-batch audit trail for free — steal that; the paper does not have it.
3. Select + inject. Concatenate the trigger lines of every skill into one short list, ask a cheap model “which ≤3 of these apply to this task,” inject the chosen bodies at the top of the prompt. Trigger-matching beats embedding search here because triggers are written to be matched. Keep the injection under ~800 tokens.
4. Reflect (failures only). One prompt over the compressed trajectory + the verifier’s actual output. Force JSON: lesson, trigger, evidence, scope_hint. Put the rejection list in the prompt verbatim — no generic advice, no basic tool usage, no replay of this specific task, nothing that names a file/repo/URL.
5. Evolve (batch level). Group candidates by topic. Show the model the current shelf, the batch’s candidates, and the remaining budget. Force one of ADD / MERGE / REVISE / SKIP. Enforce the cap in code, not in the prompt — models will happily write six items into a five-slot list.
6. Promote. Cluster candidates by failure shape; if a shape appears under two or more topics, rewrite it context-free and push it to the general shelf.

Reach for: any strong tool-use model as solver; a cheaper model as selector and evolver (the paper does exactly this and it works). Markdown + YAML front matter for storage. Git for versioning. Your existing tracing (LangSmith or equivalent) for the trajectory capture. No vector DB needed — with ten to thirty skills, LLM trigger-matching beats retrieval and is easier to debug.

The two genuinely hard parts:

• Writing a lesson that transfers. This is 90% of the difficulty and it lives entirely in the reflection prompt. The failure mode is a rule that names a file, a repo, a URL, or a customer. Mitigation: a mechanical gate that rejects any candidate containing a proper noun, path, or ID from the source task before it reaches the evolver. The paper leans on prompt instructions for this; a regex filter is cheap and strictly better.
• Deciding when a lesson is general. The paper’s answer — “it appeared in more than one topic in this batch” — is a reasonable heuristic and depends on batch size 16. With small batches you will almost never promote anything. If your volume is low, accumulate candidates across batches in a staging area and promote on a count threshold instead.

Skip for v1: the ablation apparatus, multiple solver models, REVISE (start with ADD/MERGE/SKIP), and topic auto-discovery — hand-name your five topics first.

How to Improve It

Five attackable weaknesses, roughly in order of payoff.

1. Give skills a scoreboard and let them die. The harness is write-mostly: a rule that is wrong gets injected forever and nothing measures it. Log, per skill, the pass rate of tasks where it was injected versus a held-out sample where it was not. Retire skills whose lift is negative. This is a bandit problem, and it is cheap: each skill is an arm, the verifier is the reward. It also directly attacks the Figure 6 result — the Sonnet-solver regression was probably one or two bad rules the system had no way to detect. The highest-value fix in the list.

2. Learn from near-misses, not only failures. “Failures only” is defensible but leaves signal on the table. A task that passed on the fourth attempt after three wrong turns contains a recovery procedure, and the successful trajectory shows what worked. Trigger reflection on failed OR (passed AND steps > 2× median). Cheap to test — it is one boolean.

3. Compile the stable rules into code. The paper’s own limitation section says executable skills are out of scope. Once a natural-language procedure has survived many batches, ask a model to emit it as a tool or a checklist function the agent must call — a verify_final_state() helper beats a bullet saying “verify final state,” because the harness stops depending on the solver’s willingness to comply. This also fixes the weak-solver problem: a tool executes regardless of how good the model is at following prose.

4. Test order-sensitivity and add a consolidation pass. Nobody re-shuffled the stream. Run the same task set under three seeds and measure how much the final harness and score diverge. If it diverges much, the honest fix is a periodic “sleep” pass — every N batches, re-read the whole harness with no new candidates and just merge, deduplicate, and cut. That is a maintenance step the current design lacks entirely.

5. Run the multi-agent version — the gap the authors explicitly leave open. Their limitation section says no multi-agent. But the general/topic split is begging for it: shared general skills across the whole team, topic shelves owned per role. The open question is whether a failure that happened during a hand-off between two agents compiles into a coordination rule — and who owns it. Nobody has tested whether a harness written by one role helps or confuses another. This is a real, unclaimed research direction and it is directly in your line of work.

Bonus, low effort: the missing baseline. Inject a fixed block of generic careful-agent advice (“read the failing test, verify final state, do not trust your own summary”) of the same token length as the harness, and see how much of the gain survives. If most of it does, the compiling machinery is not earning its cost. That experiment takes an afternoon and would tell you more about whether to adopt this than anything in the paper.

Glossary

• Harness — an external, editable file of guidance that shapes an agent’s behavior without changing the model. Here: markdown skill files with YAML front matter.
• Frozen solver — the agent’s LLM, whose weights never change. All learning happens in text outside it.
• Skill — one guidance entry: a trigger (when to use it), a short procedure, evidence, and a scope.
• General skill — a context-free rule promoted only after the same failure shape appears across multiple topics. Capped at 5.
• Topic skill — a localized procedure for one recurring task type, app, or domain. Capped at 5 per topic.
• Trigger — the “when does this apply” line that the selector matches against a new task; the retrieval key.
• Execution context c = (x, τ, y, f) — the raw record: instruction, action trajectory, agent’s claimed outcome, external feedback.
• Feedback grounding — whether the success signal comes from the environment (tests, verifier) or from the LLM judging itself. The paper’s key finding: self-judged is worse than not learning.
• Injection budget b — the maximum number of skills pasted into a task prompt.
• Evolver — the model role that decides ADD / MERGE / REVISE / SKIP for each candidate lesson against the current harness.
• Reflection — the model role that reads a failed execution and proposes one candidate lesson. Does not see the current harness.
• One-shot / online learning — each task is seen once, in a stream; no retries on the same task, and no offline mining pass.
• Policy (π) — in reinforcement learning, a trained mapping from state to action. Here it is borrowed notation for a prompted decision rule; nothing is trained.
• Fail-to-pass case — a task the baseline failed and the evolved system passed; used as correlational evidence, not proof that one skill caused the flip.
• SWE-bench Lite / TerminalBench-2 / τ-bench / WebArena-Infinity / CL-Bench — benchmarks for repo bug-fixing, shell tasks, customer-service tool use, stateful web navigation, and rubric-graded adaptive reasoning respectively. All five ship a programmatic verifier, which is why this method could be evaluated at all.