Toward Training Superintelligent Software Agents through Self-Play SWE-RL

TL;DR

Today’s software agents are trained on human-curated data: GitHub issues someone wrote, test suites someone maintained, “fail-to-pass” oracles someone verified. That human dependency is a ceiling — the agent mostly learns to replay human development traces, and the curation doesn’t scale. This paper removes the humans from the loop. Self-play SWE-RL (SSR) gives one LLM nothing but a Docker image of a working repo (source + dependencies, no tests, no issue text, no language hints) and has it play two roles with the same weights: a bug-injection agent that breaks the code in validated, reproducible ways, and a solver agent that has to fix it. The two roles have opposing incentives (the injector wants hard-but-solvable bugs; the solver wants to pass), which automatically generates a curriculum that tracks the model’s current ability. Trained this way on Code World Model (CWM-sft, 32B), SSR self-improves +10.4 points on SWE-bench Verified and +7.8 on SWE-Bench Pro, and beats a baseline that did get human issues and tests — across the entire training run. The headline isn’t the absolute score; it’s that an agent grounded in raw codebases can manufacture its own learning signal and surpass the human-data version.

Problem & Motivation

Here’s the concrete pain. The current recipe for a good SWE agent is: scrape GitHub for issue→PR pairs, build a Docker environment per repo, extract the test suite, identify which tests go red→green when the fix is applied, and run RL where the reward is “did the agent’s patch make those tests pass.” SWE-RL, DeepSWE, CWM, DeepSeek V3.1, Kimi K2 — they all lean on some version of this.

Three things break:

1. It’s bottlenecked by human labor. Every training instance needs a real issue, a real test, and (because the auto-extracted ones are noisy) human verification — that’s literally why “SWE-bench Verified” exists. You cannot scale human verification to the volume RL wants.
2. The agent learns to imitate, not discover. Even with RL on top, the policy is mostly refining the distribution of human development traces. It doesn’t independently discover new classes of problems.
3. Synthetic-bug shortcuts cheat. Prior synthetic approaches (SWE-smith, BugPilot) need test suites, parsers, and teacher models to distill from — so they inherit the same scalability ceiling, and they use static pipelines that ignore how good the model currently is, so the difficulty never tracks the learner.

The inspiration is AlphaZero: given only the rules of Go/chess, self-play reached superhuman play with zero human game records. The open question the authors pose: can a software agent, grounded in real repositories, learn primarily from its own interaction with code environments instead of human-curated data? Pure “zero-data” self-play (Absolute Zero, R-Zero) can’t — by introspecting against a Python interpreter you learn Python’s rules but not the vast knowledge embedded in real codebases. SSR’s bet: ground the self-play in raw repos so the agent extracts that knowledge.

What’s New (Core Contribution)

• Self-play on raw codebases with near-zero assumptions. Before: training needed issues, tests, test runners, and language-specific infra. Now: the only input is a Docker image with source + installed deps. The injection agent must discover how to run tests, write its own test parser, and figure out the suite structure entirely through tool use. This is the key move — it makes the method applicable to arbitrary repos.
• Bugs specified by a test patch, not natural language. Before: a task is a human-written issue description. Now: a bug is a formal artifact — a code-breaking diff plus a test-weakening diff whose reversal becomes the spec the solver must satisfy. No ambiguous prose to generate or grade. (They tried generating NL issues and it failed — see Unsuccessful Attempts.)
• Adversarial reward that self-balances difficulty. Before: static bug pipelines produce a fixed difficulty distribution. Now: the injector is rewarded for bugs near the solver’s frontier (hard but not impossible), and because both roles share weights and train jointly, the curriculum evolves online with the policy. Maximum injector reward is at intermediate solve rate.
• Higher-order bugs from the solver’s own failures. Before: synthetic bugs are unnatural (big code removals the solver must reconstruct). Now: when the solver fails, its broken intermediate state is recycled into a new, second-order bug — these look like the messy, layered, interdependent mistakes real developers make, and they keep the bug stream from exhausting.

What’s genuinely new vs. repackaged: the grounded self-play loop and the test-patch-as-spec formalization are the real contributions. The reward shape is adapted from Absolute Zero. The tool scaffold and RL infra are inherited from CWM. The honest framing in the paper (“a first step,” “albeit early”) is accurate — this is a proof that the loop works and beats human data, not a finished superintelligence.

How It Works (Technically)

The whole system is one LLM policy prompted into two roles, sharing parameters, updated by the same RL optimizer. Think of it as one set of weights that, depending on the prompt, either tries to break a repo or tries to fix it.

Architecture & data flow

flowchart TB
  subgraph INPUT[" "]
    DOCKER[Docker image:<br/>raw repo + deps<br/>no tests, no issues]
  end
  DOCKER --> INJ

  subgraph POLICY["Single shared LLM policy theta"]
    INJ[Bug-injection role<br/>explore, break, weaken tests]
    SOLVE[Solver role<br/>read spec, repair]
  end

  INJ --> ART[Bug artifact:<br/>bug_inject.diff<br/>test_weaken.diff<br/>test_script.sh<br/>test_files.txt<br/>test_parser.py]
  ART --> CHECK{Consistency<br/>validation<br/>incl. inverse<br/>mutation test}
  CHECK -->|fail| RINJ_BAD[r_inject = -1.0]
  CHECK -->|valid| BUILD[Build buggy codebase:<br/>apply inject + weaken,<br/>wipe .git]
  BUILD --> SOLVE
  SOLVE --> PRED[pred_patch.diff]
  PRED --> EVAL{Restore oracle tests,<br/>run test_script,<br/>parse results}
  EVAL -->|all pass| RSOLVE_GOOD[r_solve = +1]
  EVAL -->|fail| RSOLVE_BAD[r_solve = -1]
  EVAL -->|fail| HIGH[Recycle broken state<br/>into higher-order bug]
  HIGH -.-> SOLVE
  RSOLVE_GOOD --> SR[solve rate s<br/>over 8 attempts]
  RSOLVE_BAD --> SR
  SR --> RINJ[r_inject = 1 - 1+alpha times s<br/>peaks at intermediate s]
  RINJ --> UPDATE[Joint RL update to theta]
  RSOLVE_GOOD --> UPDATE
  RSOLVE_BAD --> UPDATE
  UPDATE -.online, shared weights.-> INJ

Let’s walk a single repo all the way through.

1. The injector explores and breaks. The agent gets a sandbox with just the source. Using Bash + an editor (the CWM scaffold), it pokes around: finds the test command, figures out the framework, runs the suite. Then it deliberately introduces a bug. The paper uses two strategies that produce meaty bugs rather than trivial ones:

• Removal-oriented: delete a whole method/property or code hunk (e.g., remove __eq__ from a Column class so comparison tests break), then do compatibility fixes so the project still runs. The solver now has to reconstruct missing functionality.
• History-aware: read git log, find a real past change (e.g., “Speed up fourier_series #2334”), and revert it. This recreates realistic historical bug patterns.

2. The injector packages a bug artifact — five files that formally define the bug so it can be validated and graded without any human:

• bug_inject.diff — the code change that introduces the bug.
• test_weaken.diff — a diff that removes or weakens existing tests so the bug escapes the suite (simulating a real bug that current tests miss). Crucially, reversing this patch becomes the spec: it restores the strong test, which defines exactly the behavior the fix must satisfy.
• test_script.sh — how to run the suite.
• test_files.txt — oracle test files that are reset to original before grading, so the solver can’t cheat by editing tests.
• test_parser.py — converts raw test output into a JSON map of {test_id: passed/failed}. (Language-agnostic; they just use Python.)

3. Consistency validation — the no-humans guardrail. Before a bug is “real,” it must survive execution checks: tests pass on the original code; some pass→fail after applying the bug; some failed tests pass→ again after applying the weakening patch; the bug touches a minimum number of files. The clever one is inverse mutation testing: for each file in the bug patch, revert just that file to the fixed version and re-run the strong tests. If reverting it makes a failing test pass, that file genuinely contributes to the bug. If not, the file is dead weight and the artifact fails. (Classic mutation testing checks if a test suite catches random mutations; here it’s inverted to check that each buggy file is caught by the tests.)

4. Build the buggy codebase, then solve. Apply bug_inject.diff, then test_weaken.diff, then (for higher-order bugs) a prior failed prediction. Wipe .git and re-init — otherwise the solver could read the history and “hack” the answer. The solver is handed only the reversed weakening patch (the oracle test spec) and told: make all relevant tests pass. Note: no natural-language issue is given. The downstream gain on real NL-issue benchmarks therefore comes purely from getting better at writing test-passing code, not from in-domain issue practice.

5. Grade and recycle. Apply the solver’s pred_patch.diff, restore the oracle test files (integrity), run the script, parse the JSON. All pass → solved. If the solver fails, its broken state becomes a higher-order bug and goes back into the queue (capped at second order to avoid duplicate overlap).

Demystifying the two reward functions

This is the heart. There are exactly two equations and both are simple once translated.

Solver reward (Eq. 2) — binary:

r_solve = +1  if all tests pass (pass-to-pass AND fail-to-pass)
        = -1  otherwise

Plain English: you fixed it or you didn’t. Over 8 attempts per bug, the fraction that succeed is the solve rate s ∈ [0,1]. So the expected solver reward for a bug of difficulty s is E[r_solve] = s·(+1) + (1−s)·(−1) = 2s − 1. Easier bug (higher s) → higher expected reward. The solver therefore “wants” easy bugs.

Injector reward (Eq. 1) — shaped to target the frontier:

r_inject = -1.0                  if consistency validation fails
         = -alpha                 if s == 0 or s == 1   (degenerate: too hard or trivially easy)
         = 1 - (1 + alpha) * s    if 0 < s < 1          (ideal-difficulty regime)

with alpha = 0.8. What this does: a bug that doesn’t even validate is maximally punished (−1.0). A valid bug that everyone solves (s=1) or nobody solves (s=0) gets a mild penalty (−0.8) — mild, not −1.0, so the injector still gets a usable gradient to calibrate difficulty next time. In the useful middle, reward 1 − 1.8s decreases as s rises, so the injector is paid most for bugs that are barely solvable (low but non-zero s).

The opposing-incentives trick. The solver’s expected reward 2s−1 rises with s; the injector’s reward 1−(1+α)s falls with s. Same weights, opposite gradients on the same quantity. Because the injector profits from low s but gets penalized at s=0, it’s pushed to propose bugs right at the edge of what the current solver can do. That edge moves as the solver improves — which is exactly the auto-curriculum static pipelines can’t produce.

One honest result the authors report: the solver-feedback term in the injector reward only helps slightly. A single noisy solve-rate number (estimated from 8 samples) is a weak signal — many solve rates map to similar smoothed expected rewards, so the injector struggles to learn fine difficulty control from it. But the online joint training itself — the injector running on a policy that’s continuously updated by both roles — is what produces the evolving curriculum, with or without the explicit feedback term.

The adversarial reward landscape. Drag the slider to set a bug's solve rate s. Watch the solver's expected reward (2s−1, rises with s) pull right while the injector's reward (1−1.8s in the valid middle, with −0.8 cliffs at the degenerate ends) pulls left. The equilibrium the system is driven toward is an intermediate, "hard-but-solvable" difficulty — schematic, built from the paper's Eq. 1 and 2.

The algorithm, simplified

# One shared policy `theta`. Two prompts. Joint RL. This is the SSR outer loop.
def ssr_step(repo_image, theta, alpha=0.8, n_solver_attempts=8):
    # --- INJECTION ROLE: explore the raw repo, break it, package a validated bug ---
    artifact = run_agent(theta, role="inject", env=repo_image)   # bash + editor tool loop
    if not consistency_valid(artifact):        # tests pass on orig, fail on bug, etc.
        return [(artifact, role="inject", reward=-1.0)]          # hard penalty, learn to validate

    # --- BUILD the buggy codebase the solver will see (no NL issue, no git history) ---
    buggy = apply(repo_image, artifact.bug_inject, artifact.test_weaken)
    buggy = wipe_git(buggy)                     # prevent reading the fix from history
    spec  = reverse(artifact.test_weaken)       # reversed weakening patch == the spec

    # --- SOLVER ROLE: 8 independent repair attempts, graded by restored oracle tests ---
    samples = []
    successes = 0
    for _ in range(n_solver_attempts):
        pred = run_agent(theta, role="solve", env=buggy, prompt=spec)
        ok = tests_pass(apply(buggy, pred), restore=artifact.test_files)
        samples.append((pred, role="solve", reward=+1 if ok else -1))
        successes += ok
        if not ok:
            queue_higher_order_bug(buggy, pred)  # recycle the broken state as a new bug

    # --- SHAPED INJECTION REWARD from the solve rate (the auto-curriculum knob) ---
    s = successes / n_solver_attempts
    if s in (0.0, 1.0):
        r_inject = -alpha                        # valid but degenerate difficulty
    else:
        r_inject = 1 - (1 + alpha) * s           # peaks at low-but-nonzero s
    samples.append((artifact, role="inject", reward=r_inject))

    # Both roles' (trajectory, reward) pairs update the SAME weights this step.
    return samples   # -> fed to the CWM async RL optimizer (policy-gradient style)

The RL itself is standard policy-gradient-style optimization (inherited from CWM’s async RL infra, with ScaleRL/MiniRL-flavored “large batch, low staleness” hyperparameters): sample trajectories, weight each token-sequence by its reward (advantage), push the policy toward high-reward trajectories and away from low-reward ones. Nothing exotic in the optimizer — the novelty is entirely in where the reward comes from (self-generated, validated bugs) and how the two roles’ opposing rewards shape the curriculum.

Built on Prior Work

Results & Evidence

Setup. Base model: CWM-sft (32B, pre-RL checkpoint, so RL strategies are compared fairly). Trained on 512 H100s (64 training / 448 rollout), 131K context, 16M-token global batch, 150 steps (~2.5B tokens), one attempt per eval problem at temp 1.0. Evaluated on SWE-bench Verified (500 human-verified real issues) and SWE-Bench Pro (731 enterprise-grade public problems).

Headline numbers:

• SSR self-improves +10.4 points on SWE-bench Verified and +7.8 on SWE-Bench Pro over the base model.
• SSR beats the “human-data” baseline across the entire training trajectory on both benchmarks. The baseline is the strong comparison: same hyperparameters, same environment images, but it gets the natural-language issues, the pass-to-pass/fail-to-pass tests, and the eval scripts. SSR gets none of that and still wins.
• This is despite SSR never seeing a natural-language issue in training, yet being evaluated on NL issues — so the gain is real transfer from “writing test-passing code,” not in-domain memorization.

Ablations (resolve rate over the union of 1,231 tasks):

• Self-play > parts. Full SSR beats injection-only (no repair signal at all) and repair-only (trains on a frozen set of early bugs). The evolving, online task distribution is what matters.
• Bug strategy matters. Direct/naive injection is worst — it collapses to trivial one-liners (var=0 → var=1) with no signal. Removal-only is stronger (forces reconstruction). Removal + history is best (most realistic, diverse patterns).
• Solver feedback in injector reward: marginal. Adding the solve-rate term over a consistency-only binary reward gave only a slight, “largely negligible” advantage — the signal is too noisy.

What the evidence does NOT establish (read this part if you’re selling it):

• No absolute SOTA claim. This is about self-improvement deltas and beating a matched baseline, not topping the leaderboard. Evaluation noise is ~2% paired standard error on Verified — non-trivial relative to a +7.8 delta.
• One model, one size, one base. Single 32B model plays both roles. No MoE, no separate policies, no scale curve. Their own repo-specialized and scaling attempts failed (see below).
• Unit tests are the only oracle. Real SWE is far more than “make unit tests green.” The complete oracle is in the prompt, which could invite reward hacking (not observed here, but unmitigated — no hidden test split).
• Stability ceiling. They hit training instability (“gibberish outputs”) that prevented further scaling despite using ScaleRL/MiniRL recipes. So “toward superintelligence” is aspirational; the demonstrated regime is bounded.

How You’d Use It

For an AI services shop, the directly transferable assets are the loop and the validation harness, not the 512-GPU training run.

• A self-improving QA / regression-hardening offering. The injector half is, by itself, a product: point it at a client’s repo and it discovers how to run the tests, then manufactures validated, reproducible bugs at controlled difficulty — i.e., a fuzzer that produces semantically meaningful, test-caught defects. That’s a “find the gaps in your test suite / generate a hard regression corpus” service. The inverse-mutation-testing check is the part that makes the output trustworthy (every reported bug is provably caught by a specific test).
• Eval-set generation without human labeling. The bug-artifact format (inject diff + weaken diff + script + parser + oracle files) is a clean, language-agnostic spec for an offline evaluation task. You can build a client-specific SWE-eval harness from their repo with no human issue-writing.
• Cheaper domain adaptation of a coding agent. If a client wants an agent good at their stack, you don’t need their issue tracker. Sandbox their repos, run the self-play repair loop, and the agent learns the codebase’s structure from breaking-and-fixing it. (Caveat: the paper’s own 23-repo “specialized” attempt failed for lack of diversity — so this works at corpus scale, not single-repo scale.)
• Curriculum logic for any verifiable-reward agent. The 1−(1+α)s shaped reward + opposing-incentive pairing is a reusable pattern for any generator/solver setup you run (SQL, infra-as-code, data pipelines). Whenever you can auto-verify a solution, you can auto-grade difficulty and let a generator target the frontier.

Realistic effort: the harness (consistency checks, buggy-codebase construction, parser, grading) is a few weeks of solid engineering on top of an existing tool-using agent scaffold. The RL training is the expensive, finicky part — most clients are better served by the harness + a strong off-the-shelf agent than by an actual RL run.

Build Your Own (Minimal Recipe)

A toy SSR that captures ~80% of the idea, on one machine, no RL:

1. Pick a tool-using coding agent you already have (any ReAct-style loop with Bash + file-edit tools, e.g. an open-weight code model behind your scaffold). You will prompt it into two roles rather than train two models.
2. Injector prompt → produce a bug artifact. Ask it to (a) discover and run the tests, (b) remove a meaningful code hunk or revert a git log change, (c) emit the five files. Start with removal + history — direct injection is a known dead end.
3. Build the consistency validator (the hard part #1). Deterministic Python: tests pass on original; ≥min_failing_tests go red after the bug; weakening hides them; and the inverse mutation test (revert each buggy file alone, confirm a red test goes green). This is what makes self-generated bugs trustworthy — don’t skip it.
4. Construct the buggy repo: apply inject + weaken, rm -rf .git && git init to kill the leak.
5. Solver prompt → repair. Feed only the reversed weakening patch as spec. Sample N attempts (e.g. 8). Grade by restoring oracle test files and running the parser.
6. Compute the auto-curriculum signal (the hard part #2): solve rate s → r_inject = 1−(1+α)s (with the degenerate-case cliffs). Even without RL, you can use this to select which generated bugs to keep (favor intermediate s) and to rank injector prompts — a rejection-sampling / best-of-N “self-play lite” that needs no gradient updates.
7. (Optional) Higher-order bugs: when the solver fails, snapshot its broken state and re-queue it.

Reach for: an open-weight code model (CWM/DeepSWE/Qwen3-Coder class) for the agent, Docker for sandboxing, git apply/git checkout for patch plumbing, pytest -rA + a tiny JSON parser for grading. The genuinely hard bits are (1) the consistency/inverse-mutation validator and (2) the difficulty shaping — everything else is plumbing. Skipping the RL turns this from “trains a better model” into “generates a validated, difficulty-controlled bug/eval corpus,” which is still commercially useful.

How to Improve It

1. Hidden test splits to kill reward hacking. The paper hands the solver the complete oracle. Split each bug’s tests into public (in-prompt) and private (grading-only). Now you can measure overfitting and reward genuine generalization. The authors flag this; it’s the most obvious, testable next step.
2. Separate policies / asymmetric roles. One shared 32B model plays both attacker and defender, which couples their gradients. Try a stronger or differently-tuned injector vs. solver (or an MoE where roles use different experts) and measure whether decoupling sharpens the curriculum. The authors list this as future work — it’s a clean ablation.
3. Denser difficulty signal than scalar solve rate. They found s too noisy to drive fine difficulty control. Replace the single number with structured feedback: which test categories failed, how far off the patch was (edit distance to a known fix), how many tool steps the solver burned. Exploit the generation-vs-verification asymmetry to give the injector a richer reward.
4. Beyond unit tests as the oracle. Add property-based tests, type-checks, performance budgets, or CodeClash-style goal-level verification so “correct” means more than green unit tests — pushing the agent toward real software quality, not test-passing.
5. Diversity seeding to fight mode collapse. Bugs duplicate when you sample the same repo repeatedly. Borrow Magicoder-style seeding: condition the injector on a sampled file/snippet to spread bug locations, and dedup in embedding space. Directly attacks the “23 repos = too little diversity” failure they reported.
6. Stabilize long-horizon RL. The “gibberish at scale” instability is the real blocker on the superintelligence claim. Worth attacking with the long-horizon RL toolkit (KL control, reward normalization across roles, trajectory-length curricula) — whoever solves this unlocks the scaling story.

Glossary

• Self-play — one policy improving by competing against (versions of) itself; here, the same LLM both creates and fixes bugs.
• SWE-bench Verified / Pro — benchmarks of real GitHub issues with executable tests; the agent must produce a patch that makes the tests pass. “Verified” = human-checked subset; “Pro” = harder, enterprise-grade.
• Bug artifact — the five-file formal specification of a self-generated bug (inject diff, weaken diff, test script, test files, parser) that lets the system validate and grade with no human.
• Test-weakening patch — a diff that removes/weakens tests so a bug hides; its reversal defines the behavior the solver must satisfy (the spec).
• Consistency validation — execution checks that confirm a generated bug is real and reproducible before it’s used for training.
• Inverse mutation testing — revert each buggy file alone; if a failing test goes green, that file genuinely contributes to the bug. Validates that every buggy file is detectable.
• Higher-order bug — a new bug built from the solver’s failed repair state; mimics layered, realistic developer mistakes and keeps the bug stream fresh.
• Solve rate s — fraction of (e.g. 8) solver attempts that fully fix a bug; the difficulty signal that shapes the injector’s reward.
• Reward shaping — designing the reward so the optimum lands where you want (here, intermediate difficulty) rather than at a degenerate extreme.
• Policy-gradient RL — train by sampling trajectories and nudging the policy toward high-reward ones (weighted by advantage = how much better than average a trajectory was).
• Advantage — a trajectory’s reward minus a baseline (e.g. the group mean); positive → reinforce, negative → suppress.
• CWM (Code World Model) — Meta’s 32B open-weight agentic code LLM; SSR uses its pre-RL sft checkpoint as the base and its async RL infra/scaffold.
• Pass-to-pass / fail-to-pass tests — tests that should stay green (no regressions) / tests that should flip red→green once the bug is fixed.
• Reward hacking — the agent maximizing the reward signal without doing the intended task (e.g., editing tests instead of fixing code) — mitigated here by resetting oracle test files and wiping git history.