Continual Harness: Online Adaptation for Self-Improving Foundation Agents

TL;DR

Coding agents like Claude Code work because a lot of scaffolding (tools, memory, planning prompts) is wrapped around the raw model. Nobody had built that scaffolding-builder for embodied agents (things that navigate a world step by step, seeing only part of it). This paper does two things. First, it reports “Gemini Plays Pokémon” (GPP): with a human repeatedly rewriting the scaffolding while watching, Gemini became the first AI to finish multiple Pokémon RPGs. Second, it removes the human: Continual Harness puts a “Refiner” (the same model, wearing a different hat) in charge of editing the agent’s own scaffolding every F steps, using only the trajectory so far, and never resetting the game. Starting from a bare screen-and-buttons interface with zero game knowledge, it recovers most of the efficiency gap to a hand-built expert harness — and the payoff grows with how capable the base model is. A third loop then feeds the same trajectory data back into training an open-source model, driving real in-game progress without resets.

Problem & Motivation

Here is the concrete pain. A frontier vision-language model, dropped into Pokémon with just a screenshot and eight buttons, makes almost no progress. The PokeAgent Challenge showed this directly. The model can reason, but it walks into walls, forgets what it was doing, loops in menus, and loses long battles.

The industry fix is a harness: the scaffolding layer between the model and the world. Claude Code, OpenHands, and OpenClaw are harnesses for coding — they give the model file tools, shell access, memory, and planning prompts. That scaffolding is why those agents feel competent. But for embodied, long-horizon, partially-observable tasks (you only see part of the world, and the payoff is thousands of steps away), the equivalent harness has to be hand-built by an expert for each game: A* pathfinders, a type chart, a damage calculator, curated objective lists. That is expensive, brittle, and doesn’t transfer.

Two half-solutions existed and both fall short:

• Hand-engineered harnesses (the expert route) work but require a human domain expert per environment. No automation.
• Prompt-optimization methods (GEPA, MIPRO) automate part of it, but they only rewrite the system prompt, and they need episode resets: run a full episode, score it, tweak the prompt, start over. That is impossible when the interesting failures only show up 100,000 steps in, and impractical for any real agent (a coding agent or an ops agent) where “just reset the environment” is costly or meaningless.

The gap: automatic, full-harness, mid-episode, reset-free self-improvement for embodied agents. Nobody had it. GPP proved a human could do it by hand. This paper automates the human away.

What’s New (Core Contribution)

Four contributions, precisely:

1. GPP as an existence proof (empirical). Before: frontier VLMs make near-zero RPG progress. Now: with human-in-the-loop harness refinement, Gemini completed Pokémon Blue, Yellow Legacy (hard mode), and Crystal — the first AI system to finish multiple Pokémon RPGs. Crucially, in the hardest stages the model itself started editing its own strategy through long-context memory. That emergent behavior is the seed for the rest of the paper.

2. Continual Harness, a reset-free full-state refiner (the method). Before: prompt optimizers rewrite only the prompt p, and reset between updates. Now: a Refiner rewrites the entire harness state — prompt, sub-agents, skills, memory — via CRUD edits (create/read/update/delete), mid-episode, from a partial trajectory window, with no reset. This is the actual novelty: it generalizes “optimize the prompt across resets” to “rewrite the whole scaffold as you go.”

3. A capability-dependent gain, measured (the empirical characterization). Before: “better scaffolding helps” was folklore. Now: on the Emerald cost-vs-completion plane, the harness benefit scales with model capability: strictly dominant on Gemini 3 Pro (~40% cheaper at full completion), high-variance on Flash, and below a capability floor on Flash-Lite (it actually hurts). This is an honest, quantified boundary condition.

4. Model–harness co-learning, closing the loop (the training pipeline). Before: the harness shapes behavior at inference only. Now: an open-source model’s rollouts through the live-refining harness are scored by a process reward model, low-reward windows are relabeled by a frontier teacher, and the model’s weights are updated by soft SFT — reset-free, iteration over iteration. The refined harness shapes the trajectories; the trajectories surface new failures for the next refinement. Both loops run on the same data.

How It Works (Technically)

There are two nested loops. The inner loop is a normal agent step. The outer loop is the Refiner rewriting the harness. A separate training loop wraps both when you want to update model weights, not just the scaffold.

The four things the harness holds. The harness state is written H = (p, G, K, M):

• p — system prompt: strategic instructions handed to the model every step.
• G — sub-agents: specialized modules the orchestrator can hand off to (a battle strategist, a puzzle solver, a self-reflection pass). Handing off gives the sub-agent a small, focused context, which is cheaper per step.
• K — skills: reusable routines. Two flavors — text heuristics the model cites in reasoning, and executable programs (a BFS pathfinder, a tool wrapper). Some ship with the harness (press_buttons, get_game_state); others get authored during play.
• M — memory: a persistent store of facts, strategies, and observations that survives across the trajectory.

The model edits all four through a fixed set of meta-tools: define_agent, run_code, process_memory, and friends. That is the important trick — the model doesn’t get pre-built game tools; it gets tools for building tools.

The inner loop (one agent step). Write the observation s_t = (o_t, m_t): a rendered frame o_t plus an ASCII text map m_t (walkable tiles ., walls #, NPCs N, the player’s position). The map is there because VLMs are bad at fine spatial reasoning over pixels; it contains no walkthrough and no objectives. The agent picks an action:

$$a_t \sim M(\cdot \mid s_t, H_t, \tau_t)$$

In plain English: the action a_t (a list of button presses) is sampled from the model M, conditioned on the current observation s_t, the current harness H_t, and the trajectory so far τ_t. There is no separate “policy network” — the policy is the model reading its own scaffolding. The action goes to the emulator, the game advances a fixed number of frames, and you get s_{t+1}.

The outer loop (harness refinement). Every F steps, after a warm-up of W steps, the Refiner — the same model M, just invoked with a different job — reads the recent window τ_{t-F:t} and looks for failure signatures: navigation loops, tool-call failures, stalled objectives, missed exploration. Then it runs four passes, one per component:

1. Rewrite the prompt p given the failures it found.
2. CRUD the sub-agents G: create entries for repeated multi-step patterns, edit entries to fix detected failures, delete entries that never got used productively.
3. Codify skills K from sequences that worked; repair executable code that threw exceptions.
4. Update memory M: add entries to fill gaps, refresh stale ones, demote importance for areas the agent has moved past.

It emits a bundle of edits ∆ = (∆p, ∆G, ∆K, ∆M), and the harness updates:

$$H_{t+1} = H_t \oplus \Delta$$

⊕ just means “apply the CRUD edits.” p gets replaced by ∆p; G, K, M get create/update/delete operations. The agent does not reset. The new harness enters the agent’s context on the very next step.

Why reset-free matters, stated as the paper’s core claim: refinement information accumulates monotonically. A failure seen at step 5,000 stays available to every later refinement pass, so refinement quality compounds with episode length. Reset-based methods throw that accumulation away every time they restart — and they can never even reach failures that only appear deep in an episode (late-game battles, multi-step puzzles), because each iteration resets to the start.

Architecture & data flow

flowchart LR
  subgraph Env[Environment]
    E[(Pokémon emulator)]
  end
  subgraph Harness[Harness state H = p,G,K,M]
    P[System prompt p]
    G[Sub-agents G]
    K[Skills K<br/>text + executable]
    MEM[Memory M]
  end
  E -->|frame o_t + ASCII map m_t| AG[Agent = model M]
  Harness -->|context| AG
  AG -->|action a_t = buttons| E
  AG -->|trajectory tau| BUF[Trajectory window]
  BUF -->|every F steps| REF[Refiner = model M]
  REF -->|CRUD via meta-tools| Harness

flowchart TD
  S[Observe s_t = frame + text map] --> ACT[Model picks action a_t]
  ACT --> STEP[Emulator advances 120 frames]
  STEP --> LOG[Append to trajectory tau]
  LOG --> CHK{t mod F == 0<br/>and t > W?}
  CHK -->|no| S
  CHK -->|yes| READ[Refiner reads window tau_t-F:t]
  READ --> SIG[Detect failure signatures:<br/>loops, tool errors, stalls]
  SIG --> FOUR[4 passes: rewrite p; CRUD G; codify/repair K; update M]
  FOUR --> MERGE[H = H merge delta]
  MERGE --> S

Schematic of the two-loop mechanism. The inner loop (agent → environment) runs every step; every F steps the Refiner reads the recent window and applies CRUD edits, so the harness state on the right grows and gets repaired without the game ever resetting. Illustrative, not the paper's logged data.

The training loop (co-learning). Everything above adapts the scaffold. To also adapt the weights of an open-source model, the paper wraps the whole thing in an online loop (Figure 2b in the paper):

Each iteration k runs the current policy π_{θ_k} inside a live-refining harness for K=256 steps (the harness keeps editing itself the whole time). Then:

1. A pairwise process reward model (PRM) R(s_t, a_t, τ) ∈ [0,1] scores each transition over a sliding window. Reward is a weighted mix: trajectory progress 0.4, action correctness 0.3, reasoning quality 0.2, format compliance 0.1.
2. Low-reward windows get relabeled by a frontier teacher (Gemini-3.1-pro) — the teacher supplies what the student should have done there.
3. A soft SFT update on that relabeled shard produces the next checkpoint:

$$\theta_{k+1} = \theta_k - \eta \nabla \mathcal{L}_{\text{SFT}}$$

That is a plain supervised-learning gradient step: nudge the weights θ to make the teacher’s relabeled actions more likely (η is the learning rate; the LoRA setup uses 3 epochs at 5×10⁻⁶). The loop is reset-free: the saved emulator state at the end of iteration k becomes the start of iteration k+1, so the model’s in-game position accumulates across its own training.

The subtle point: the data distribution D_θ the model trains on depends on θ through the harness. The model’s actions produce trajectory τ; the Refiner reads τ and rewrites H; H shapes the next observations. Weights and scaffold co-adapt.

The algorithm, simplified

# Continual Harness: the reset-free inner+outer loop (one continuous episode).
# llm(role, ctx) -> text/action.  A single model M plays both "agent" and "refiner".
# H is the mutable harness: prompt p, sub-agents G, skills K, memory M.
# meta_tools apply CRUD edits to H in place (define_agent, run_code, process_memory, ...).

def continual_harness(env, H, F=100, W=500, steps=200_000):
    tau = []                                    # trajectory grows monotonically, never reset
    s = env.reset()                             # the ONLY reset — start of the episode
    for t in range(steps):
        # --- inner loop: normal agent step, conditioned on the CURRENT harness ---
        a = llm("agent", context(s, H, tau[-K_CTX:]))   # a = list of button presses
        s, done = env.step(a)                            # env advances; partially observable
        tau.append((s, a))

        # --- outer loop: refine the WHOLE harness from the recent window, no reset ---
        if t > W and t % F == 0:
            window = tau[-F:]
            failures = diagnose(window)         # nav loops, tool errors, stalled objectives
            dp = llm("refiner", rewrite_prompt(H.p, failures, window))
            dG = crud_subagents(H.G, window, failures)   # create repeated patterns, delete dead ones
            dK = codify_and_repair_skills(H.K, window)   # new skills from wins; fix code that threw
            dM = update_memory(H.M, window)              # add/refresh/demote entries
            H = apply(H, dp, dG, dK, dM)         # H = H ⊕ Δ, enters agent context next step
    return H                                     # the harness itself is the transferable artifact

The single idea to take away: the agent and the thing that improves the agent are the same model, and the improvement is written into a persistent scaffold that never gets wiped. That is what “continual” and “reset-free” mean here.

Built on Prior Work

Results & Evidence

GPP (the headline demo). First AI to complete Blue (May 2025), Yellow Legacy hard mode (Aug 2025), and Crystal (Nov 2025) — the latter without losing an end-game battle. The Elite Four in Yellow took ~18-20 lifetime attempts per opponent before a clean run. The harness updates were concentrated and recurrent, not one-and-done: a small set of navigation and battle components got repeatedly rewritten across 200k+ turns. This is a real, unusually hard demonstration, but it is a demo with a human in the loop — quality is established by the completion record, not a controlled baseline.

Continual Harness vs. minimal vs. expert (the controlled result). On Red and Emerald, across Gemini 3 Pro/Flash/Flash-Lite, HCH starts from the bare interface and substantially cuts button-press cost versus the minimal harness and recovers a majority of the gap to the hand-built expert — with no decompilation, no milestone schedule, no hand-built sub-agents. Most of that gain is carried by the skill library alone (BFS/A* pathfinder wrappers), because saved navigation presses translate directly into faster milestones.

Capability-dependent gain (the honest boundary). On the Emerald cost-vs-completion plane:

• Pro: strictly Pareto-dominant. From-scratch HCH hits 100% of milestones at a $130 median vs. minimal at 98% for $215 — ~40% cheaper, no completion loss.
• Flash: high variance. Bootstrap-updating reaches 80% at $42 vs. minimal 77% at $30 — marginal.
• Flash-Lite: below the capability floor. Minimal reaches 20% at $11; every HCH variant falls to 3-13% at equal or higher cost. The harness actively hurts a model too weak to use it.

Cost vs. completion on Emerald, rebuilt from the paper's reported numbers. The benefit of self-refinement grows with model capability: on Pro the refined harness moves up-and-left (more milestones, less cost); on Flash-Lite it moves down-and-right, below the minimal baseline — the capability floor.

Skills measurably self-improve toward an oracle. Scoring evolved navigation skills against a Dijkstra shortest-path oracle, the path-cost deficit falls from a ~50% penalty to single digits early and stays there — and the repair happens in the same episode as the failure. This is the cleanest evidence for the reset-free claim.

The evolving navigation skills close on a Dijkstra shortest-path oracle within one run: path-cost deficit (how much longer than optimal, lower is better) drops from ~50% toward the 0% oracle line, because the Refiner diagnoses and repairs failing skills mid-episode. Schematic of the paper's Figure 8 trend.

Co-learning on open-source models. A Gemma-4 26B student, warmed up with SFT + offline GRPO (neither of which alone moves milestones), then run through the online DAgger+PRM loop, shows sustained in-game milestone progress across training iterations — from both early and mid-game checkpoints, reset-free. A cross-family Qwen3.5 negative control, without the SFT warm-up, produces parseable tool calls but cannot leave the starting area — ruling out a protocol artifact.

What the evidence does NOT establish:

• No convergence. They report sustained progress over the horizon they ran; they never hit a plateau or completion in the co-learning loop.
• Teacher is a frontier model. The open-source loop depends on Gemini-3.1-pro relabeling. This is distillation, not self-play. The authors admit Gemma-4 (≤31B) isn’t strong enough to be its own teacher yet.
• Reset-free vs. reset head-to-head is open. They study only the reset-free regime; no controlled comparison to batch-with-resets on the same task.
• Reuse is thin and fragile. Memory reference rates are low; most authored entries sit unused. On Red, bootstrap-updating agents sometimes abandon inherited sub-agents (inherited share collapses to ~6%) and the milestone staircase regresses below even the minimal baseline.
• Narrow domain. Everything is Pokémon RPGs on an emulator. Generalization to coding/ops (where they argue reset-free matters most) is asserted, not shown.

How You’d Use It

For an AI services company, the transferable idea is not “play Pokémon.” It is: let a Refiner rewrite an agent’s own scaffolding from its logs, continuously, without restarting the job. Where that slots in:

• Self-tuning agents for clients. Most production agents you ship get manually tuned when they fail — someone reads the logs, edits the system prompt, adds a tool. Continual Harness is that job, automated: a scheduled Refiner pass over the last N steps that CRUD-edits the prompt, the tool set, and a memory store. You can sell “agents that debug their own scaffolding” as a differentiator.
• Long-running ops and coding agents. These are exactly the reset-free regime the paper argues for — you can’t “reset the production environment.” A Refiner that codifies a working shell recipe into a reusable skill, and repairs a tool that started throwing, mid-run, is directly valuable.
• Skill-library-as-moat. The paper’s own finding is that the harness, not the episode, is the transferable unit. The compounding asset is the accumulated, repaired skill/sub-agent/memory library. For a services firm, that library per client (or per vertical) is a durable, hard-to-copy asset.
• A capability-tiering rule for pricing. The capability floor is a business fact: refinement only pays off above a model-quality threshold. Run it on your strongest model (Pro-tier); do not ship it on a cheap model — it will spend more and complete less. That maps cleanly onto “premium autonomous tier vs. basic scripted tier.”

Realistic effort: a useful v1 (prompt + skill + memory refinement on a logged agent) is a few engineer-weeks. The weight-training co-learning loop is a research project, not a client deliverable.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value — skip the weight training entirely; the inference-time refiner is where most of the gain lives.

Components:

1. An agent loop you already have: observe → llm(prompt + tools + memory) → act. Log every (state, action, reasoning) to a trajectory buffer.
2. A harness object with four mutable fields: prompt (string), subagents (dict of name→spec), skills (dict of name→code/heuristic), memory (list of entries with IDs).
3. Meta-tools the agent and refiner both call: define_agent, run_code/define_skill, edit_memory, edit_prompt. These are the CRUD API over the harness.
4. A Refiner function: every F steps, feed the last F trajectory entries to the model with a prompt like “Here is what the agent did. Find repeated failures. Emit edits to prompt/subagents/skills/memory as JSON CRUD ops.” Apply the ops. Don’t reset.

Build order:

1. Get the bare agent + trajectory logging working (Hmin).
2. Add the memory field + a Refiner that only writes/updates memory. Verify it helps.
3. Add executable skills with run_code + a repair pass (re-run failing skills, feed the traceback back to the Refiner). This is where most of the paper’s gain came from — prioritize it.
4. Add sub-agent handoffs last (cheapest per-step context, but the most fragile).

The one or two genuinely hard parts:

• Failure diagnosis over a window. “Detect a navigation loop / a stalled objective / a schema-mismatched tool call” from raw logs is the crux. The paper’s own worst failure (the 1,003-turn Power Plant loop, where the agent repeated a broken tool call 842 times while believing it was progressing) is exactly a diagnosis failure. Budget for a good, explicit failure-signature prompt and a sanity check that the environment actually changed.
• Safe self-authored code. run_code that writes and executes new skills needs a sandbox and a schema the meta-harness enforces, or you get silent no-ops (the paper’s schema-mismatch bug) and worse.

Libraries/models to reach for: any strong tool-calling model for both roles (the paper uses one Gemini 3 model as agent and refiner — you don’t need two models). A JSON-schema-validated tool layer. A simple key-value or vector store for memory. For skills, a restricted Python sandbox.

How to Improve It

Limitations as leverage — concrete, testable:

1. Add a reuse prior / deletion policy for sub-agents. The paper’s biggest regression (Red bootstrap-updating collapsing below the minimal baseline) is caused by the agent abandoning inherited, already-repaired sub-agents in favor of unrepaired new ones. A rule that deprecates a newly authored component whose task signature is already covered by an inherited one — or a soft prior that biases selection toward battle-tested components — is a direct, testable fix the authors flag themselves.

2. Close the diagnosis blind spot with a “did the world change?” guard. The Power Plant loop happened because the agent trusted its own reasoning over the environment. A cheap environmental-delta check (has the game state actually moved in N steps?) that force-triggers a Refiner pass would catch confident stalls. Test: measure time-to-recovery on injected stall scenarios.

3. Replace the frontier teacher with self-generated targets. The co-learning loop currently distills from Gemini-3.1-pro. Try group-relative self-improvement (GRPO-style: sample several rollouts through the harness, reinforce the better ones by the PRM) so a single open model can improve without a stronger teacher. Test: does milestone progress hold when the teacher is removed?

4. Establish reset-free vs. reset head-to-head. The paper explicitly leaves this open. Run the same task under (a) reset-free state propagation and (b) batch accumulation with resets, same compute. This would turn the central claim from “compelling story” into “measured advantage.”

5. Regularize the create-and-forget tail. Most authored skills/memories are never used again. Add a lightweight utility score per component (invocations × success) and let the Refiner garbage-collect or merge low-utility entries, keeping the harness lean and the per-step context cheap. Test: does completion hold while context length drops?

6. Transfer the harness across environments, not just runs. They show the harness transfers across episodes of the same game. Test whether a harness refined on Red gives a head start on Emerald — if the abstractions (pathfinding, battle triage) transfer, that is a much bigger moat than per-game scaffolds.

Glossary

• Harness — the scaffolding layer wrapped around a model: prompt, tools, sub-agents, memory. What turns a chatbot into a competent agent.
• Embodied agent — an agent that acts step by step in a world it perceives partially (here, a Pokémon game via screen + buttons), as opposed to answering a one-shot question.
• Partially observable — the agent can’t see the full state (NPC intent, hidden battle mechanics); only what the frame and map expose.
• Reset-free — the environment is never restarted; the agent’s position and its scaffold accumulate continuously across the whole run (and across training iterations).
• Refiner — the same model, invoked with a different job: read the recent trajectory, find failures, and rewrite the harness.
• CRUD edits — create / read / update / delete operations; here, how the Refiner changes sub-agents, skills, and memory entries.
• Meta-tools — tools for building tools (define_agent, run_code, process_memory): the model uses them to edit its own harness in place.
• Sub-agent — a specialized module the main agent hands off to (battle strategist, puzzle solver) to get a smaller, cheaper, focused context.
• Skill — a reusable routine, either a text heuristic cited in reasoning or an executable program (like a BFS pathfinder) the agent can author during play.
• Trajectory (τ) — the logged sequence of states and actions so far; the raw material the Refiner reads.
• Failure signature — a recognizable bad pattern in the trajectory: navigation loop, tool-call error, stalled objective, missed exploration.
• Milestone metric / button-press cost — the benchmark scores progress by canonical in-game milestones reached, and cost by cumulative button presses (so batching [A, A, DOWN] into one tool call still counts as three presses; rewards efficient action).
• Pareto-dominant — better on every axis at once; here, more milestones and lower cost than the baseline.
• Capability floor — the model-quality threshold below which self-refinement can’t bootstrap and actually hurts (Flash-Lite falls below the minimal baseline).
• SFT (supervised fine-tuning) — training the model to imitate given (input → correct output) examples; here, from teacher-relabeled trajectories.
• LoRA — low-rank adapters: a cheap way to fine-tune a large model by training small added matrices instead of all weights.
• GRPO (group-relative policy optimization) — an RL method that samples several answers per prompt, normalizes their rewards within the group to get an advantage, and reinforces the above-average ones. No separate value network.
• Advantage — how much better an action’s reward is than the baseline (here, the group mean); high-advantage actions get reinforced.
• PRM (process reward model) — a model that scores each step of a trajectory (not just the final outcome), giving denser training signal.
• DAgger — an imitation-learning method where an expert/teacher relabels the states the student actually visits, fixing the distribution mismatch of plain imitation.
• Soft SFT — a gentle supervised update (few epochs, tiny learning rate) on the relabeled shard, used here as the per-iteration weight update.
• Text map — an ASCII grid derived from emulator memory showing walkable tiles, walls, NPCs, and the player’s position, to compensate for VLMs’ weak pixel-level spatial reasoning.