TL;DR
Agents fail in new software because they don’t know its quirks: how a table shifts its columns, what “paid” really means on a booking page, which preservation rule actually allows moving a box. RSIAgent learns those quirks without any training. A curriculum agent writes practice projects near the target task, an actor agent does them by submitting Python or Bash programs, and an isolated verifier agent checks the real result and returns PASS or FAIL. After each verdict the actor writes what it learned into a folder of plain files. Practice runs in two stages: broad (many different projects in parallel waves) and then deep (one harder project at a time, aimed at whatever just failed). Then the memory is frozen and handed to the actor for the real run. With GLM-5.3 as actor and Kimi-K3 as verifier and curriculum, partial score goes from 71.97 to 78.98 on OSWorld 2.0 and from 83.75 to 84.82 on Agents’ Last Exam. The largest jump over frontier models comes from the actor-verifier harness itself, not from the memory. And the “frozen” memory is built per task, around that exact task.
Problem & Motivation
You deploy an agent into a tool it has never really seen. Maybe it’s a CAD package, a niche audio editor, an internal booking site, or a WPS spreadsheet. The model knows roughly how software works. It doesn’t know this software: the column that resizes when a cell says FREE, the timeout banner that shows up even though the payment went through, the clipboard read that comes back empty because the page hasn’t finished drawing.
You have two usual fixes, and both are bad here:
- Fine-tune on more interaction data. This is expensive. It usually needs humans to make demonstrations. And you can’t do it for a private or fast-changing environment.
- Stuff past successful trajectories into context. This is cheap, but it stores what worked once, not why. Reused out of its original setting, a memorized trajectory breaks.
The paper wants a third option. The agent goes and learns the environment itself and writes down cause and effect (“if the columns are content-sized, re-detect them after every state change”). Nobody labels anything, and no weights change.
What’s New (Core Contribution)
- A three-role practice harness with hard information boundaries. Before: single-agent self-reflection (Reflexion-style), or self-improvement loops that need benchmark scores to know whether a change helped (DGM, Hyperagents). Now: curriculum, actor and verifier each get a separate context. The verifier never sees the actor’s reasoning or memory. The curriculum agent sees only a throwaway copy of memory and can’t edit the real one. Only the actor writes memory. This separation is the real design contribution. It’s aimed squarely at correlated errors, where the agent that did the work also grades it.
- Broad-then-deep exploration (BRS → DRS). Before: Voyager-style automatic curricula, which keep expanding coverage. Now: two stages with different shapes. BRS runs parallel waves of varied “prerequisite, variant, contrast and stress” projects, and every project in a wave starts from the same memory snapshot. DRS is sequential and gets harder each round. It attempts the target, studies the failure, and builds practice that tells competing explanations apart. The authors compare this to pretraining followed by post-training. That’s a nice framing, but the mechanism is just coverage first, then targeted debugging.
- Memory as actor-owned, reconciled files. The memory has no schema. After each verdict the actor runs two passes. Distill pulls out procedures, constraints and failure lessons. Reconcile checks the whole memory bank for contradictions, stale assumptions and claims broader than the evidence supports. This is an incremental idea, but the reconcile pass is the part most memory systems skip.
- Honest failure taxonomy. Three ways the loop goes wrong: exploration that never tests the weak belief, a verifier that approves an incomplete answer, and a local PASS that hardens into an over-broad rule. This is the most reusable part of the paper.
What’s not new: the code-as-policy action space (CodeAct), automatic curricula (Voyager), experience memory (AutoManual, ExpeL-style), and LLM verifiers. The contribution is how these parts are assembled and fenced off from each other.
How It Works (Technically)

The formal setup, demystified
The paper writes the agent as
a_t ~ π_θ(· | q, h_t, M) and (s_t, o_t) ~ E(· | s_{t-1}, a_t)
In plain English, the next action a_t is something the model π_θ samples, conditioned on three things: the task q, this episode’s history h_t (observations and actions so far), and a durable memory M. The environment then moves to a hidden state s_t and returns an observation o_t.
What this does operationally: θ (the weights) never changes. h_t is wiped between tasks. M is the only thing that survives. So all “self-improvement” means making M better. The whole paper is a strategy for writing M.
An action is a complete program. The actor can’t click. It submits python3 or bash, gets back stdout+stderr and the exit code, and repeats. It also has look (ask for a screenshot description), ask (query a simulated user, when one is configured) and done.
The three roles
| Role | Sees | Can do | Cannot do |
|---|---|---|---|
| Actor (GLM-5.3) | task, environment, execution output, current memory | run programs, write memory after a verdict, write a “learning diagnosis” | grade itself |
| Verifier (Kimi-K3) | task requirements, candidate environment | inspect from a QEMU checkpoint (restored afterward); return PASS / FAIL / UNVERIFIED | see actor reasoning or memory; edit memory |
| Curriculum (Kimi-K3, separate context) | target query, past outcomes, actor’s diagnosis, a disposable copy of memory | write practice projects with input fixtures; decide continue / stop | grade candidates; edit canonical memory |
Two details matter more than they look:
- The verifier inspects a snapshot, not the live machine. The harness saves a full QEMU checkpoint and restores it after inspection. So the verifier can poke at things (open files, run tests) without changing what gets graded. If you build a verifier that can act, you need this.
- UNVERIFIED is a third verdict, and it teaches nothing. If the evidence can’t settle the question, the outcome is not recorded as a success or a failure. That keeps ambiguous outcomes out of memory.
Architecture & data flow
flowchart LR Q[Target query q] --> CUR[Curriculum agent] CUR -->|practice project + fixtures| ACT[Actor agent] MEM[(Canonical memory M<br/>actor-authored files)] -->|read| ACT ACT -->|python / bash programs| ENV[Environment VM] ENV -->|stdout, exit code, screenshots| ACT ACT -->|candidate artifact| VER[Verifier agent] ENV -->|checkpoint copy| VER VER -->|PASS / FAIL + findings| ACT ACT -->|distill + reconcile| MEM ACT -->|learning diagnosis| CUR MEM -.->|disposable copy| CUR
The two stages
flowchart TD
M0[Empty memory] --> W[BRS wave: curriculum writes up to 4 independent projects]
W --> P[Run all in parallel from the SAME memory snapshot]
P --> V[Verifier: PASS or FAIL per project]
V --> U[Actors resume one at a time, in curriculum order, and update memory]
U -->|budget left and not SATURATED| W
U -->|budget hit or SATURATED| T[DRS: attempt the real target]
T --> TV{Target verdict}
TV -->|FAIL, or PASS with doubt| PR[Curriculum writes ONE harder or contrastive practice]
PR --> PV[Actor runs, verifier judges, actor updates memory]
PV --> T
TV -->|PASS and no more useful practice| F[Freeze memory, hash it]
F --> E[Final run: actor + verifier, no learning, official grader scores]
Stage 1, Broad Recursive Self-exploration (BRS). The curriculum agent gets the exact target query, but only as a direction. It’s told not to attempt the unchanged target yet. It writes a wave of projects around the target’s “capability neighborhood”: prerequisites, variants, contrasts and stress cases. Every project in a wave starts from the same frozen pre-wave memory and can’t see its siblings, so they are really independent. When the wave finishes, the actors resume one at a time, and each writes into the latest memory, including updates from earlier siblings in the same wave. This is the “parallel work, sequential updates” rule. It avoids merge conflicts in a schemaless memory without needing a merge algorithm. The budget is a nominal 8 projects with up to 4 running at once, and it’s checked only between waves. The curriculum can end the stage early by declaring SATURATED.
Stage 2, Deep Recursive Self-exploration (DRS). Now the actor attempts the real target, with the environment reset first. After the verdict, the curriculum agent reads the outcome and the actor’s learning diagnosis. After a FAIL, it writes a project that separates plausible causes. After a PASS, it adds practice only if some belief looks over-generalized. Each practice is verified and saved to memory before the next decision. Any practice forces another target attempt. DRS ends when the target passes and the curriculum sees nothing more worth learning. Or it ends with STALLED, followed by one last attempt.
Test time. The memory is frozen and its file-tree hash is recorded. The environment is reset. The actor and verifier run on the target with no writeback, and an external official grader scores the result. That grader never talks to the learning agents.
One trace: the railway booking task (T065)
- BRS, project 8. The curriculum isolates one skill: watch a seat map that changes over time, and find the first release where one D seat and one F seat in the same row are free together. The actor’s fixed pixel grid misreads releases 3 and 4. The cause is the table’s column widths, which depend on content: a column showing FREE is about 72 px wide, an occupied one about 36 px. The actor changes its method and re-detects the columns in every state, using the wider aisle gap as an anchor. Its background watcher also failed to compile twice, so the actor switched to an inline capture loop (663 frames over about 335 s). The verifier checks the frames against the release schedule and returns PASS. Memory grows from 10 files / 70,513 bytes to 12 files / 91,697 bytes, including the line “a frozen grid misclassified releases #3/#4 until columns were re-detected per state”.
- DRS, cycle 2. The actor books two Paid trains, but the verifier FAILs it because the segments don’t connect. The actor records a stricter transfer rule.
- DRS, cycle 3. The actor re-reads the task, which explicitly allows boarding at an intermediate stop. It books G102 + G122, and the verifier PASSes. Memory now holds both the failure and the refined condition. A new lesson also gets added: a scripted clipboard read can come back empty while the page is still rendering, so check that the read is complete before concluding there’s no inventory.
- Test time. The final actor reads the booking, GUI-capture and form-filling playbooks in its first two iterations, then picks a different itinerary (G98 + D756) that fits the live interface. The score goes from 0 to 1. One caveat, which the authors flag themselves: this run also got an explicit clarification of the current date that the baseline didn’t get.
Step through the broad-then-deep lifecycle. In BRS, four projects run in parallel from the same memory snapshot, then commit to memory one at a time. In DRS, one harder project at a time is aimed at the last failure. The memory bar grows with each commit. Byte counts follow the paper's T049 run (89 KB after BRS, 141 KB at freeze). The step timing is schematic.
The algorithm, simplified
# llm(role, prompt) -> str ; run_program(env, code) -> (stdout, exit_code)
# Memory is a dict of filename -> text. No schema. Only the actor writes it.
def practice(env, memory, project):
env.reset(project.fixtures)
snapshot = dict(memory) # actor works against a fixed view
trajectory = actor_solve(env, project.instruction, snapshot)
ckpt = env.checkpoint() # verifier may poke; graded state must not change
verdict = llm("verifier", f"Task: {project.instruction}\nInspect env. PASS only if EVERY "
f"material requirement is affirmatively supported.")
env.restore(ckpt)
return trajectory, verdict
def learn(memory, trajectory, verdict):
if verdict.label == "UNVERIFIED":
return memory, None # ambiguous outcomes teach nothing
draft = llm("actor", f"Distill causal lessons; keep uncertainty.\n{trajectory}\n{verdict}")
memory = llm("actor", f"Reconcile ALL memory against this draft: fix contradictions, "
f"narrow over-broad rules, drop stale claims.\n{memory}\n{draft}")
diagnosis = llm("actor", "What seems reliable, what is uncertain, what test would tell?")
return memory, diagnosis
def rsi(env, target, budget=8, width=4):
memory, outcomes, done = {}, [], 0
while done < budget: # ---- BRS: parallel waves
wave = llm("curriculum", f"Near {target}, NOT it: {width} diverse projects", outcomes)
if wave == "SATURATED": break
results = parallel_map(lambda p: practice(env, memory, p), wave) # same snapshot
for traj, v in results: # sequential commits, curriculum order
memory, diag = learn(memory, traj, v); outcomes.append((v, diag))
done += len(wave)
while True: # ---- DRS: one deeper step at a time
traj, v = practice(env, memory, target)
memory, diag = learn(memory, traj, v)
nxt = llm("curriculum", f"After {v.label}: one project that discriminates causes, or READY", diag)
if v.label == "PASS" and nxt == "READY": break
if nxt == "STALLED": break # (paper: one final attempt, then stop)
if nxt != "READY":
memory, _ = learn(memory, *practice(env, memory, nxt))
return freeze(memory) # hashed; no writeback at evaluation
The new part sits in three places. practice keeps the verifier blind to the actor and protects the graded state with a checkpoint. learn has the actor reconcile the whole memory rather than append to it. The two while loops have different shapes on purpose: wide and parallel first, then narrow and sequential.
Built on Prior Work
| Prior idea | What it gave | What this paper changes |
|---|---|---|
| CodeAct (code as action) | Programs as the action unit, revised with execution feedback | Uses it for GUI apps too; the actor never clicks |
| Voyager (automatic curriculum) | An LLM picks the next skill to practice | Adds the broad-then-deep split and a target-conditioned curriculum that stops when expected value runs out |
| Reflexion / AutoManual / experience memory | Verbal lessons persisted across attempts | The lesson writer is never the grader; adds a reconcile pass over the whole memory |
| DGM, HGM, Hyperagents | Agents that modify themselves, judged by benchmark scores | No benchmark score or label in the loop; an LLM verifier grounded in environment evidence instead |
| GEPA / prompt evolution | Optimize prompts from reflective feedback | Optimizes a free-form memory folder, not a prompt, and never sees the score being optimized |
| HyMEM, EchoTrail-GUI, ZhuLong, CoEvoSkills | Structured memories and exploration for computer-use agents | Schemaless actor-owned files; the novelty is role separation and staging, not memory structure |
| Play2Code (game playtesting loop) | Iterative playtest-and-revise | Same loop plus a curriculum and memory; holds up on strong base games where Play2Code regresses |
Results & Evidence

Main table (partial score %, binary in parentheses):
| System | OSWorld 2.0 (82 tasks) | ALE Near-term (67 tasks) |
|---|---|---|
| Kimi-K3 alone (leaderboard) | 58.30 (—) | 71.60 (40.30) |
| Claude Opus 5 | 70.19 (34.72) | 79.54 (46.27) |
| GPT-6 Astra | 72.60 (—) | 82.26 (52.24) |
| RSIAgent harness, no RSI | 71.97 (37.80) | 83.75 (49.25) |
| RSIAgent full | 78.98 (42.68) | 84.82 (50.75) |
Where the gain comes from. Each bar stacks the model alone, then what the actor-verifier harness adds, then what RSI memory adds, next to GPT-6 Astra (dashed line). Toggle between benchmarks. The numbers are the paper's Table 1. Kimi-K3 alone is the leaderboard figure for the verifier/curriculum model; the actor is GLM-5.3.
Other evidence:
- Stage ablation (4 OSWorld tasks): the mean partial score is 74.54 with both stages, 65.52 broad-only, and 56.50 deep-only. Broad-only beat the baseline on every task. Deep-only fell below the baseline on two of four. Starting deep without broad coverage can hurt, which supports the broad-then-deep ordering.
- Per-task trajectories: T044 (video editing) reaches 100%, T049 (presentation repair) 80%, T065 (railway booking) 100%, all by step 8. Gains come in jumps: a score sits flat until the last bottleneck is resolved.
- Game development (40 GameCraft-Bench tasks): every base generator improves. With GLM-5.3-Flash as the base, the overall score goes 30.55 → 38.25 (Play2Code) → 44.73 (harness without memory) → 48.72 (full). When the base game is already strong (Codex + GPT-5.5), Play2Code makes it worse (52.77 → 51.05), while RSIAgent still lifts it to 61.28.
- Memory size: a few hundred KB of plain files per task. T044 ends at 316,541 bytes, T049 at 141,312 bytes, T065 at 281,085 bytes.
What the evidence does NOT establish. Read these before you quote the headline:
- Most of the win over frontier models is the harness, not the self-improvement. Without any RSI, the GLM-5.3 + Kimi-K3 actor-verifier harness already scores 83.75 on ALE, which beats GPT-6 Astra (82.26). On OSWorld it gets 71.97, within a point of GPT-6 Astra’s 72.60. RSI adds about 7 points on OSWorld and about 1 on ALE. The real lesson is “two open models in a code-as-policy actor-verifier loop are frontier-class”, and RSI adds on top of that.
- The memory is per task and conditioned on the target, not a general environment memory. The curriculum agent sees the exact target query in both stages. DRS attempts the target itself again and again, with feedback from the internal verifier (not the official grader), before the final “frozen” run. So this is test-time practice on the test task, with a reset before scoring. It is not “explore an app once, then reuse the memory across unseen tasks”. Transfer across tasks is never measured.
- Only some tasks got RSI. On OSWorld, 41 of 82 tasks have an RSI score, and the rest keep their baseline score. On ALE it’s 19 of 67. Tasks already at full credit were skipped, so the gains are concentrated where there was room. That’s fine for the aggregate, but don’t read it as “RSI on every task”.
- The comparison isn’t matched. Competitor numbers are copied from leaderboards, each with its own harness and budget. The authors say so in Appendix C.4. Exploration compute is large and not reported as one total: practice runs can take up to 2,000 iterations and 24 hours each.
- Binary accuracy doesn’t follow partial score. On ALE, RSIAgent’s binary score (50.75) is below GPT-6 Astra’s (52.24). The extra partial credit doesn’t always turn into finished tasks.
- The ablation is small. It uses four tasks chosen from a cohort that had already improved, and the “full RSI” bars average two runs.
How You’d Use It
Your harness. This is where most of the value is, and you can take it without any of the exploration.
- Where it slots in: split the agent that does the work from the agent that accepts it. Give the verifier the task and the artifact, and hide the actor’s reasoning and notes. Give it three verdicts (PASS, FAIL, UNVERIFIED) and the instruction “PASS only when every material requirement is affirmatively supported; falsify nearby plausible substitutes”. If the verifier can act (run tests, open files), give it a snapshot or a container copy, never the live state.
- What it improves: per the paper’s own table, this harness alone took Kimi-K3 from 71.60 to 83.75 on ALE when paired with GLM-5.3. That’s the cheapest upgrade in the paper.
- Cost: about one extra model call per attempt, plus some sandbox plumbing for snapshots.
- Must be true: the result has to be checkable from the artifact and the environment (files, rendered output, saved state). A verifier that only reads the actor’s summary brings back the correlated error.
Your automations and business processes. Put a “practice before production” phase in front of any unattended agent that runs in a tool it doesn’t know well: a new SaaS admin console, a CAD export step, a booking portal, an ERP screen.
- Where it slots in: on a staging copy, let a curriculum agent write 8 drills around the real job (for example, “export a STEP file with these holes”, “read a seat map that changes”). Run them, verify them, and let the actor write a playbook folder. Ship the frozen playbook with the production agent, and keep its hash in your audit log.
- What it improves: edge cases you’d otherwise discover in production: dynamic layouts, misleading “timeout” banners, half-rendered pages.
- Cost: real compute (hours of agent time per workflow), plus you need a resettable staging environment.
- Must be true: the job repeats enough to pay back the practice, and the environment is stable enough that last week’s lessons still hold.
Your workflows and methodologies. The failure taxonomy works as a review checklist for any memory or “lessons learned” system you run, agentic or human:
- Did the practice actually test the belief you’re unsure about, or just the easy neighbors? (This was 3 of 4 exploration failures.)
- Did the verifier check the full requirement checklist, or only that “an output exists”? (Fidelity gaps were 3 of 6 verification failures.)
- Does each stored rule carry its conditions and doubt, or did a local PASS become a universal rule? (Rule-scope loss was 2 of 3 memory failures.)
Your applications. If your product has agents acting inside user-specific software (a user’s own CRM setup, a private internal tool), BRS/DRS gives you an onboarding mode. On first connect, the agent practices on a sandbox copy and builds a per-workspace playbook. Charge the compute once and reuse the playbook across that user’s tasks. Know that the paper never tests reuse across tasks, so measure that yourself before you count on it.
Build Your Own (Minimal Recipe)
Smallest useful version (one day of wiring):
- Sandbox with reset + snapshot. Use a Docker container (or VM) per project, with
reset(fixtures)andcheckpoint()/restore(). This is the boring part, and it’s what makes everything else trustworthy. - Actor loop. A model that only acts by submitting a complete
python3/bashprogram and gets back stdout+stderr and the exit code. Cap the iterations. Adddoneas a structured response. - Verifier. A different context (ideally a different model) that gets the task text and shell access to the checkpointed sandbox, and returns
{verdict: PASS|FAIL|UNVERIFIED, findings}. Never pass it the actor’s transcript. - Memory folder. Plain markdown files plus an
INDEX.md. After each verdict, the actor runs two prompts in its own context: distill (P5) and reconcile (P6). The paper’s prompt excerpts in Appendix B are short and worth copying almost word for word. - Curriculum. Start with BRS only: one call that returns 4 JSON projects (
id,instruction,fixtures) around the target, with “do not attempt the target itself”. Run 2 waves. That alone beat deep-only exploration in the paper’s ablation. - Add DRS later. Attempt the target, feed the verdict and diagnosis back, get one discriminating project, and repeat until PASS + READY or a hard cap.
Libraries and models. For the sandbox, docker SDK or e2b. For the GUI, pyautogui/xdotool/Playwright, called from the actor’s programs. Any two decent tool-calling models work. Use a different model for the verifier than for the actor to reduce correlated errors. The paper splits GLM (actor) and Kimi (verifier/curriculum).
The two hard parts:
- Memory concurrency. Parallel projects must start from one snapshot, and their commits must be applied one at a time. Skip this and you’ll get silent overwrites or contradictory rules in a schemaless folder. The paper’s fix is simple and correct: resume each actor’s context in a set order, each one seeing the latest memory.
- Verifier quality decides everything downstream. A false PASS doesn’t just inflate one score. It becomes a rule in memory that later runs trust. Invest here first: derive a requirement checklist from the task, require evidence for each item, and use UNVERIFIED freely.
How to Improve It
- Measure cross-task transfer. Explore one environment once with a held-out set of target tasks, and the curriculum never sees them. Then evaluate on them. This is the claim the title implies and the experiments don’t test. If the score holds, the method is far more valuable. If not, you’ve learned it’s per-task test-time search.
- Give memory claims a confidence and scope field. Rule-scope loss and unenforced uncertainty caused all three memory failures. Don’t impose a full schema. Just require each rule to carry
observed_in:andconfidence:lines, and have reconcile lower confidence when a rule is reused and the run then fails. This is testable by comparing the failure-mode counts. - Make the curriculum target uncertainty, not coverage. Most exploration failures were target-skill mismatch. Have the actor’s diagnosis list its 3 least-trusted beliefs explicitly, and score each proposed project by whether it would falsify one of them. That’s a cheap active-learning signal.
- Use more than one verifier for consolidation. Use one verifier for the actor’s repair loop and a second, independent one before any PASS becomes memory. Let disagreement default to UNVERIFIED. This targets the “accepted mistake becomes a rule” path directly.
- Report the compute bill and a matched baseline. Give the same harness the same token budget as pure best-of-N retries on the target, with no memory. If RSI doesn’t beat N blind retries at equal compute, the memory isn’t adding much. This one experiment would settle the biggest open question.
Glossary
- Code-as-policy — the agent acts only by writing and running programs, never by clicking. GUI control happens inside those programs.
- Curriculum agent — the role that decides what to practice next and when to stop. It never grades or writes memory.
- Actor agent — the role that does the work and owns the memory.
- Verifier agent — an isolated role that checks the real result and returns PASS / FAIL / UNVERIFIED.
- BRS (Broad Recursive Self-exploration) — parallel waves of varied practice projects around the target, all starting from the same memory snapshot.
- DRS (Deep Recursive Self-exploration) — sequential, harder-each-round practice aimed at what just failed, with repeated target attempts.
- Frozen memory — the memory folder after exploration, hashed and read-only for the final run.
- Distill / reconcile — the actor’s two learning passes. Distill pulls lessons from one episode. Reconcile fixes the whole memory against them.
- Learning diagnosis — the actor’s short note to the curriculum agent: what seems reliable, what’s uncertain, and what test would tell them apart.
- SATURATED / STALLED / READY_FOR_TARGET — the curriculum’s stop signals: nothing more worth learning; can’t write a useful project; go back to the target.
- QEMU checkpoint — a full VM snapshot taken before the verifier inspects, then restored so inspection can’t change the graded state.
- Partial score / binary accuracy — average task score (0 to 100%) versus the share of tasks with full credit.
- OSWorld 2.0 / Agents’ Last Exam (ALE) — benchmarks of long, real software workflows (desktop apps, web services, professional deliverables).
- Training-free — no weights change. All improvement lives in context and memory.