Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback

TL;DR

Giving a tool-using LLM agent a good written “skill” (a short procedural doc for a task family) reliably boosts its success rate. But when an LLM tries to write or fix that skill itself, agents using the LLM-authored skill actually do worse than agents given no skill at all — following instructions and authoring good instructions are different capabilities. WER (Write, Execute, Refine) closes that gap by training a dedicated policy, the Skill Optimizer, to do nothing but revise skills, while the agent that actually executes the skill (the “executor”) is never touched. The optimizer proposes several candidate rewrites of a skill, a frozen executor tries each one for real in a sandboxed tool environment, and a deterministic, code-based checker — not a model — decides whether the task actually succeeded. Training uses GRPO to compare candidate skills against each other (not in isolation) and deliberately keeps the “half worked, half didn’t” cases — a matched success trajectory and a failure trajectory from the same skill and task — as training material for the next round, so the optimizer learns from the consequences of its own earlier writing. Across two multi-turn tool-use benchmarks, this lifts pass rates 4–8 points over no skill and beats the same executor paired with every off-the-shelf frontier model used as a skill-writer instead — despite the trained optimizer being only 4B parameters.

Problem & Motivation

Tool-using agents do better when they’re handed a “skill”: a short markdown cheat-sheet describing how to handle a class of tasks — which tool to call when, how to validate arguments, what to check before retrying. On SkillsBench, expert-written skills raise average pass rate from 33.9% to 50.5%. That’s the promise.

The problem is authorship. When the skill is written by an LLM instead of a human expert, agents using it score 8–11 points below using no skill at all. The paper’s framing: following a procedure and converting execution evidence into a correct procedural fix are not the same skill, and nothing in ordinary pretraining or instruction-tuning specifically trains the second one.

The obvious fix — an inference-time loop where an LLM drafts a skill, watches it run, and edits it (Reflexion-style self-correction) — only patches the current document. The model doing the diagnosing never gets better at diagnosing; every new task starts from the same untrained judgment. And these loops are risky in practice: injecting just 10% plausible-but-wrong “lessons learned” into an agent’s experience dropped τ²-bench Pass@1 from 82.5 to 77.2, and adding a self-verification pass barely recovered any of it (83.3 → 83.2). A model checking its own homework doesn’t reliably catch its own mistakes.

So the real question the paper asks: can the ability to turn execution evidence into a better skill be trained, the way tool-use itself gets trained — rather than re-invoked from a fixed model’s in-context reasoning every single time?

What’s New (Core Contribution)

• A dedicated, trainable Skill Optimizer, decoupled from the executor. Before: skill-writing was either a human expert or an ad hoc call to a fixed general-purpose LLM inside a workflow. Now: a separate policy is optimized end-to-end, by RL, specifically for the task “write a better skill given what happened last time” — while the agent that runs the skill never changes.
• A deterministic, programmatic reward instead of a model-graded one. Before: most self-improvement loops score their own output with another LLM call (or the same model), reintroducing exactly the untrustworthy judgment they’re trying to fix. Now: success is decided by comparing final environment state to a ground-truth reference — code, not opinion.
• Phase-wise self-bootstrapping via mixed-outcome retention. Before: skill-refinement RL methods (SkillMaster, SkillOS, Skill-R1) learn that skills can be improved by RL, but each treats one round of revision fairly generically. WER’s specific mechanism: after scoring K candidate skills × n rollouts each, it keeps only the candidates that were exactly half right — one rollout succeeded, one failed, same skill, same task — and hands that matched pair to the next training phase as the thing to fix. This deliberately manufactures the training curriculum from the optimizer’s own near-misses.
• Group-relative credit assignment for a coarse, binary-ish reward. Rather than trying to build a fine-grained reward (was the right tool picked? was the argument correct?) — which would require a model-in-the-loop judge again — WER accepts a coarse verifier and instead compares multiple candidates written for the same situation, so a consistent gap between two skills is informative even when each individual score is coarse.

How It Works (Technically)

The object being optimized. A skill is a short markdown doc with four fixed parts: a name, a one-line description of the task family, a numbered workflow, and a notes section for edge cases and known failure modes. At run time this text is just prepended to the executor agent’s system prompt — nothing else about the agent changes. The skill has to describe a procedure, not a solution: it can’t bake in specific IDs or values, because those belong to one task instance and wouldn’t transfer.

The refinement state. Everything the optimizer is allowed to see when it writes a new skill is packaged into one object:

x = (q, C, h, s, e)

• q — the user’s query for this task
• C — the tool definitions / environment description visible to the agent
• h — the interaction history so far
• s — the skill currently in force
• e — the execution evidence from the last time s was actually run

The Skill Optimizer is a policy over documents: s' ~ π_θ( · | x ) — given that bundle, sample a new skill. This one equation is the entire generation rule in the framework — there’s no separate “write from scratch” mode. A cold start is just the special case where s is a zero-shot draft and e is the evidence from that draft’s first run. Every later round is the identical operator with different inputs, which is exactly why training it once lets you apply it repeatedly at inference time.

Two channels reach the optimizer, and they carry different information.

1. The trajectory, as context (uncompressed). When a candidate skill s' is handed to the frozen executor π_A, it produces a full call-by-call trajectory: tool calls, their arguments, what the environment returned (including errors), and the terminal state. This is not summarized before it’s shown to the optimizer next round — the raw sequence is what tells the optimizer where to edit. A reward of zero only tells you the skill failed; a trajectory that shows the same doomed API call issued three times against an ID the environment already invalidated tells you the skill never told the agent to check state before acting.
2. The outcome, as reward. Whether s' actually helped is decided by the environment, not a model: for BFCL, the final state of every environment object is compared against the reference solution’s final state; for τ²-bench, the terminal database is compared to a reference database and required user-facing actions are checked. Both checks are code, run once, deterministic. This is the load-bearing design choice in the paper: freezing the executor keeps the optimizer’s own outputs out of the trajectory being judged, and using a programmatic verifier keeps a model out of the score. Either one alone isn’t enough — you need both to avoid a self-grading loop.

The reward, concretely. For a candidate skill s', scored over n executor rollouts:

R(s') = ( R_fmt + R_task + R_len ) / 3

• R_fmt — is the output parseable (separate reasoning block vs. skill body, so only the body gets injected downstream)? This term is annealed down during training once the model reliably gets the format right, so it stops competing with the term that actually matters.
• R_task — the verifier’s pass/fail outcome, averaged over the n rollouts. This is the term doing the real work.
• R_len — penalizes empty reasoning or reasoning that blows past a length budget.

Why compare candidates instead of scoring one in isolation (GRPO in plain terms). Absolute reward is dominated by task difficulty, not by whether a given skill is well-written — some tasks are just harder than others regardless of instructions. So instead of learning “reward 0.7 is good,” the optimizer generates K candidate skills for the same refinement state and compares them to each other:

Â_k = R_k − mean(R_1, ..., R_K)          # advantage: how much better than its siblings

This is the group-relative part of GRPO (Group Relative Policy Optimization). Instead of training a separate value-function network to estimate a baseline (as classic PPO does), GRPO just uses the mean of a group of candidate outputs sampled for the same input as the baseline. The advantage tells the optimizer “make this kind of output more likely” (positive Â) or “less likely” (negative Â) — relative to its own siblings, not relative to some global scale.

The actual parameter update is the clipped GRPO surrogate:

J(θ) = E_x [ (1/K) Σ_k min( ρ_k · Â_k,  clip(ρ_k, 1−ε, 1+ε) · Â_k ) ]

where ρ_k = π_θ(s'_k | x) / π_old(s'_k | x) — the ratio of how likely the new policy is to produce this candidate versus how likely the old policy (before this update) was. In plain terms: nudge the model toward candidates that scored above the group average, away from ones that scored below it, but clip the size of that nudge so one batch of noisy rollouts can’t yank the weights too far in one step (this is the same trust-region trick PPO uses). Two things are notably absent here that show up in most RLHF setups: there’s no learned value function (GRPO’s whole point is to avoid training one), and there’s no KL penalty pulling the policy back toward its starting point. The authors drop the KL term deliberately: they want the model to learn a document style and revision behavior the base model doesn’t already have, so anchoring it to its own initialization would fight the objective.

Why the reward stays coarse on purpose. It’s tempting to break R_task into finer credit — was the right tool picked, was the argument correct, did the agent recover after an error. In these environments none of that can be checked programmatically without putting a model back in the loop, which is exactly the failure mode the paper is trying to avoid. So the verifier is only allowed to answer what it can answer for certain (did the final state match), and the finer differences are recovered structurally — through the group comparison and the trajectory channel — rather than through a more granular score.

The credit-assignment trick that makes cross-phase training work: mixed-outcome retention. Every scored candidate — its skill text, all n trajectories, and its reward — is written to an experience buffer. With n = 2 rollouts and a binary-ish verifier, a candidate scores 0, 1, or 2 out of 2. Only the score-of-1 candidates are kept for the next phase. Why: a candidate that succeeded on both rollouts leaves nothing to diagnose; one that failed on both leaves no working path to contrast against. A 1/2 candidate is a matched pair — same skill, same task, same frozen executor, two branches, one that reached the reference state and one that didn’t. Because everything except the branch taken is held fixed, the difference between the two trajectories is, structurally, exactly the edit that’s missing from the skill. The next refinement state is built by literally concatenating this pair (success block + failure block + the skill that produced them) rather than summarizing it — the optimizer reads the same call sequences the verifier actually scored.

Phase-wise self-bootstrapping. One round of revision only closes whichever gap the last execution happened to expose; the next gap is invisible until the revised skill is run again. So WER trains in phases: phase p’s retained mixed-outcome records become phase p+1’s refinement states. This grows what the paper calls a revision tree — a retained skill from round t, plus its evidence, becomes the parent from which round t+1’s K candidates are sampled. Practically, this shifts the training distribution over time toward skills that are “almost but not quite sufficient” — precisely the state where one targeted edit is most likely to flip the outcome, and precisely the state a static, pre-collected dataset couldn’t supply (whether a skill is “almost sufficient” is a moving fact about the current optimizer paired with the current executor).

Architecture & data flow

flowchart LR
  OPT["Skill Optimizer<br/>(trainable policy π_θ)"] -->|"writes skill s′"| AGENT["Frozen Executor Agent π_A"]
  subgraph SB["Execution Sandbox"]
    AGENT -->|"tool calls"| ENV[("Stateful Environment")]
    ENV -->|"observations / errors"| AGENT
  end
  AGENT -->|"trajectory τ (verbatim)"| OPT
  AGENT -->|"trajectory τ"| VER["Programmatic Verifier<br/>(code, not a model)"]
  VER -->|"pass / fail reward"| OPT

Why GRPO compares candidates to their own group instead of scoring them on an absolute scale. Toggle between an "easy" and a "hard" refinement state: the raw pass rates shift a lot with task difficulty, but the group-relative advantage (how much each candidate beats or trails its own siblings) stays a meaningful, comparable signal either way — that's what actually drives the policy update.

One phase of training, and how it feeds the next

flowchart TD
  X["Refinement state x<br/>(query, tools, history, skill, evidence)"] --> SAMPLE["Sample K candidate skills<br/>s′₁ ... s′_K ~ π_θ(·|x)"]
  SAMPLE --> EXEC["Execute each candidate:<br/>n rollouts on frozen executor"]
  EXEC --> SCORE["Verifier scores each rollout<br/>R = (R_fmt + R_task + R_len)/3"]
  SCORE --> ADV["Group-relative advantage<br/>Â_k = R_k − mean(R)"]
  ADV --> UPD["GRPO update π_θ"]
  SCORE --> RETAIN{"Candidate's n rollouts:<br/>mixed outcome?"}
  RETAIN -->|"1 success + 1 fail"| BUFFER["Pair success+fail traj.<br/>into next refinement state"]
  RETAIN -->|"all pass or all fail"| DISCARD["Discard — nothing to diagnose"]
  BUFFER --> X2["Phase p+1 refinement states"]
  X2 -.->|"repeat, one more revision round"| X

A revision tree across phases. Each node is one candidate skill scored over 2 rollouts (green = both passed, red = both failed, gold = mixed — exactly the ones WER retains). Only gold nodes spawn the next phase's children; a skill that already works everywhere, or nowhere, is a dead end for training.

A worked example (from the paper’s case study). A BFCL task: read a financial report, average revenue/expense/profit, round it, write it to a new file. The seed skill under-specifies both file creation and the aggregation math — both rollouts fail (0/2). Revision 1, having seen a trajectory where echo failed because the file didn’t exist yet, adds an explicit “create with touch before writing” instruction — the file-operation error disappears, but one rollout still averages the numbers incorrectly (1/2, a retained mixed-outcome case). Revision 2, built from that exact matched pair, adds “aggregate all values before applying mean; round only the final result” — and now both rollouts pass (2/2). Each revision fixes exactly the failure the previous round’s evidence exposed, not a generic list of best practices.

The algorithm, simplified

# One phase of WER training: propose skills, execute, score, update, and
# build the next phase's training data from what actually happened.

def train_phase(states, optimizer, executor, env, K=4, n=2):
    buffer = []                                    # (state, skill, trajectories, reward)

    for state in states:                           # state = (query, tools, history, skill, evidence)
        candidates = optimizer.sample(state, k=K)   # K rewritten skill docs for this ONE state
        rewards = []

        for skill in candidates:
            trajectories = [executor.run(skill, env, state) for _ in range(n)]
            reward = score(skill, trajectories)      # format + task-success + length
            rewards.append(reward)
            buffer.append((state, skill, trajectories, reward))

        # group-relative advantage: how much better/worse than its siblings --
        # NOT an absolute score, which mostly measures task difficulty
        mean_r = sum(rewards) / len(rewards)
        advantages = [r - mean_r for r in rewards]
        optimizer.grpo_update(state, candidates, advantages)   # clipped surrogate, no KL term

    next_states = []
    for state, skill, trajectories, reward in buffer:
        successes = [t for t in trajectories if t.passed]
        failures  = [t for t in trajectories if not t.passed]
        if successes and failures:                  # mixed outcome only -- the informative case
            next_states.append(make_refinement_state(state, skill, successes[0], failures[0]))
    return next_states                              # becomes the input to the NEXT phase


def score(skill, trajectories):
    fmt    = well_formatted(skill)                   # parseable reasoning + skill blocks
    task   = mean(t.passed for t in trajectories)    # verifier outcome, deterministic
    length = within_budget(skill)                    # penalize empty or bloated reasoning
    return (fmt + task + length) / 3

Built on Prior Work

Results & Evidence

Two multi-turn, tool-using benchmarks, both with programmatic (code-based) verifiers: BFCL v4 multi-turn (200 tasks — File System, Trading, Travel, Vehicle domains; 50 train / 150 test) and τ²-bench (Airline, Retail, Telecom; same 1:3 split). Metric is Pass@1 (single attempt, since a skill is meant to help an agent succeed the first time), averaged over 3 evaluation runs. The frozen executor for all training and evaluation is GPT-4o; the trained Skill Optimizer is Qwen3-4B.

Headline numbers (Table 1):

• WER beats No Skill by 7.80 points (BFCL) and 3.85 points (τ²-bench) — the gap the paper opened with is closed and reversed.
• WER beats the untrained GPT-5.1 seed skill by 9.35 and 3.02 points — so the gain isn’t just “GPT-5.1 writes good first drafts,” it’s the refinement.
• Training is doing the work, not just the workflow. Swap the trained Qwen3-4B optimizer for the same untrained Qwen3-4B model in the identical refinement loop, and it doesn’t improve over the seed skill on BFCL (67.28% either way) and actively hurts τ²-bench (47.70% → 40.43%, worse than no skill). A refinement pipeline alone, without a trained diagnoser, is not sufficient — and can backfire.
• A trained 4B model beats untrained frontier models in the same role. Using GPT-5.5, DeepSeek-V4-Flash, Gemini 3.5 Flash, or Claude Sonnet 4.6 as the (untrained) skill optimizer in the identical workflow tops out at 74.75% (GPT-5.5). WER’s trained 4B model reaches 76.63% — 1.88 points higher than the best off-the-shelf model, despite being far smaller. Specialized training on this specific capability beats raw general reasoning ability.
• Phase-wise bootstrapping matters, not just more optimization steps. Evaluating the checkpoint after each training phase: 69.35% (phase 1) → 71.29% (phase 2) → 76.63% (phase 3), monotonically improving — evidence that later phases teach something later phases specifically supply (harder, self-generated refinement states), not just “more gradient steps on the same data.”
• Refinement gains saturate — and can reverse. At inference time, running more rounds of revision on top of the GPT-5.1 seed: depth 0 (seed) 67.28% → depth 1 70.67% → depth 2 76.63% → depth 3 75.33%. Almost all the gain happens in the first two rounds; a third round makes things slightly worse.

BFCL v4 average Pass@1 as more WER refinement rounds are run at inference time (Table 3). Gains front-load into the first two rounds, then flatten and dip slightly — a practical answer to "how many rounds should I run in production."

Caveats the paper is explicit about, and some worth flagging yourself:

• Both benchmarks have clean, programmatic verifiers by construction. Nothing here establishes how WER performs where success can’t be checked in code — which is most real-world business processes.
• Test sets are small (150 BFCL, ~113 τ²-bench tasks) with results averaged over 3 runs; the per-domain numbers (especially τ²-bench’s individual domains) should be read with real uncertainty bars in mind, even though the paper doesn’t report them.
• WER doesn’t win everywhere: it exactly ties No Skill on BFCL’s Trading domain, and Trace2Skill (a non-RL baseline) beats WER on that same domain (75.68% vs. 70.27%).
• The pipeline leans on proprietary frontier models at two points that aren’t trained or ablated in isolation: GPT-5.1 generates the zero-shot seed skill, and GPT-5.5 merges multiple refined skills into one before evaluation. How much of the final number depends on those two fixed, expensive calls (versus the trained 4B optimizer) isn’t cleanly separated.
• Only one frozen executor (GPT-4o) is tested. The paper’s own theoretical argument — that a difference between two candidate skills is evidence “about the two documents” only because the executor is fixed — is unverified across executors; it’s plausible but untested whether a skill trained against GPT-4o transfers cleanly to a different executor.
• The paper’s own stated limitation: trajectories are kept verbatim across phases, so refinement-state size grows with interaction length — untested on longer-horizon, multimodal, or large-skill-repository settings where this could become a real context/cost bottleneck.

How You’d Use It

If you’re building or operating agentic tool-use systems — API integrations, back-office workflows, CRM/ERP automations — this paper is describing exactly the failure mode you’ve probably already hit: writing an SOP/runbook for an agent by hand works, but auto-generating or auto-fixing that SOP with an LLM tends to make things worse, not better, unless someone is watching closely. WER is a recipe for turning that “someone watching closely” into a trainable, repeatable process instead of a one-off prompt-engineering pass.

Where it slots in:

• Your harness — a playbook that gets better on its own. Instead of shipping a static agent plus a hand-tuned system prompt, you can run continuous refinement of your agent’s operating instructions against its actual tool environment, using real pass/fail evidence rather than another model’s opinion. The durable part isn’t the base model — it’s the training/verifier pipeline you build around it.
• Your harness — a retrofit for an agent you can’t or don’t want to touch. Because the executor is frozen throughout, this is a way to improve an agent’s effective behavior (via the skill it’s handed) without retraining, fine-tuning, or even having write access to the underlying model — useful when the executor is a third-party API-backed model, a vendor’s deployed system, or something you’re not allowed to modify directly.
• Your automations — the prerequisite, and the hard part, is the verifier. This only works where you can programmatically check “did the task actually finish correctly”: final database state matches expected, file exists with the right content, invoice reconciled, ticket closed with the right resolution code. Building that check is the valuable engineering work here; most of your own processes don’t have it out of the box, and it’s worth treating as its own project before you invest in the RL layer on top.
• Your workflows — a cheap way to test the idea before committing to training. The “propose K, execute each, keep the score, feed forward” loop is valuable even with an untrained, prompted model doing the writing — it’s just capped lower and, per the paper’s own ablation, can actively hurt if the writer isn’t specifically trained. Use the untrained version to validate your verifier and sandbox setup are solid before investing in the RL step.

Build Your Own (Minimal Recipe)

Stage 0 — prompted-only version (no training, captures the mechanism, not the compounding gains).

1. Define a skill schema: name, one-line task-family description, numbered workflow, notes/edge-cases. Match this paper’s structure — it’s simple and it works.
2. Stand up a sandboxed replica of your tools/APIs (mock or staging) so tasks can be re-run safely and repeatedly — this is required for both the multiple rollouts per candidate and honest A/B comparison.
3. Write a deterministic verifier per task family: a state-diff check against a known-good reference outcome. This is the hard, valuable part — resist the temptation to let an LLM grade itself here; that’s the exact failure mode the paper is built to avoid.
4. Implement the loop: propose K candidate skill rewrites with any capable LLM → run each n times against the frozen executor in the sandbox → verifier scores each → keep the skill from the best-scoring candidate (or, better, the mixed-outcome case) → feed it back in as the next round’s starting skill. No training required yet — just inference budget and a working verifier.

Stage 1 — add real training once Stage 0 is validated. 5. Swap the untrained prompted writer for a small, trainable model (Qwen3-4B-class) and wrap the same loop in GRPO, using verl or a comparable RL framework (TRL, OpenRLHF). Use the paper’s reward: format compliance (anneal it down once it’s reliable) + task success (the term that matters) + a length term. 6. Drop the KL-to-reference term — you want the model to diverge from its starting behavior toward a document-revision style it doesn’t have yet. 7. Implement the retention rule: with n=2 rollouts per candidate, keep only the score-of-1 (mixed-outcome) cases as the seeds for the next phase. Concatenate the success and failure trajectories verbatim with the producing skill — don’t summarize. 8. Run multiple phases; each phase’s retained records become the next phase’s training states. Expect most of the value in the first two or three phases (matches the paper’s own diminishing-returns finding). 9. Optionally add a merge step at inference (a single prompted call to consolidate several refined skill variants into one canonical skill per task family) if you want one document to ship rather than a family of candidates.

The two genuinely hard parts: (a) the verifier — building one that’s actually trustworthy without a model in the loop is most of the engineering effort and the main thing separating this from a self-confirming loop; (b) the sandbox — you need a tool environment safe and cheap enough to re-run the same task many times per training step, which for real production APIs usually means building a faithful staging replica, not just pointing at prod.

How to Improve It

1. Test it where verification isn’t free. Real workflows rarely have a clean programmatic ground truth. Try training a lightweight, human-labeled verifier from a small seed set, bootstrap WER on top of it, and measure how much verifier noise it takes before the self-confirmation problem this paper is designed to avoid creeps back in.
2. Fix the context-growth problem the paper flags itself. Verbatim trajectories accumulate across phases. Try compressing older phases while keeping the newest phase verbatim, or retrieving just the point where the paired success/failure trajectories first diverge instead of the full transcript — measure how much diagnostic signal survives.
3. Add model selection across refinement depth instead of a fixed round count. Depth-2 beats depth-3 on average (76.63% vs. 75.33%) — a sign of overfitting to rollout noise at deeper rounds. Validate each round’s merged skill on a small held-out slice before deploying it, rather than always running exactly two rounds.
4. Use richer retention than “exactly half.” With only n=2 rollouts, retention is a hard binary rule. With n=4 or more, outcomes span a real distribution (1/4, 2/4, 3/4) — worth testing whether weighting retained candidates by closeness to 50/50, rather than requiring an exact split, yields more and better training signal.
5. Stress-test the “frozen executor = clean attribution” assumption. Train against one executor, evaluate the resulting skills against a different one (or several, simultaneously, during training) to check whether the learned revisions are genuinely about the skill document, or partly overfit to GPT-4o’s specific quirks.

Glossary

• Skill — a short markdown “how-to” document (name, description, numbered workflow, notes) prepended to an agent’s system prompt to guide it on a class of tasks.
• Executor / executor agent — the LLM agent that actually calls tools and does the task, guided by whatever skill it’s given. Frozen (never trained) throughout WER.
• Skill Optimizer — the separate, trainable policy whose only job is to read a skill plus what happened when it ran, and write a better skill.
• Refinement state — the bundle (query, tool context, history, current skill, execution evidence) the optimizer conditions on to write its next revision.
• Rollout — one full run of an agent on a task, start to terminal state; used loosely to mean “one execution attempt.”
• Trajectory — the complete, ordered record of one rollout: every tool call, its arguments, the environment’s response, and the final state.
• Verifier — here, a deterministic piece of code (not a model) that checks whether a rollout’s final state matches a known-correct reference outcome.
• Policy — in RL, the (here: language) model being trained, viewed as a function from a situation to a distribution over actions — in this paper, from a refinement state to a distribution over possible new skill documents.
• Reward — a scalar score assigned to an action (here, a candidate skill, after being executed) that RL training tries to increase.
• Advantage — how much better or worse a specific action’s reward was compared to some baseline; the actual quantity RL uses to decide which direction to push the policy.
• GRPO (Group Relative Policy Optimization) — an RL method that estimates the advantage baseline as the mean reward of a group of candidates sampled for the same input, instead of training a separate value-function network. Removes the need for a value model; makes the reward “relative,” canceling out per-task difficulty.
• Group-relative — scored against siblings sampled for the same situation, not against a fixed or global scale.
• Clipped surrogate objective / PPO-clip — a training-stability trick that caps how much a single update is allowed to shift the policy’s probability of an action, to prevent one noisy batch from causing a destructive jump in behavior.
• KL penalty / reference policy — a common RLHF regularizer that penalizes the trained policy for drifting too far (in KL-divergence terms) from its starting point. WER deliberately omits this because it wants the model to learn genuinely new behavior.
• Credit assignment — the general RL problem of figuring out which part of a long sequence of actions (or a long document) deserves the blame or credit for an eventual outcome.
• Cold start — the very first round for a task, where the “current skill” is a zero-shot draft rather than a previously revised one.
• Phase — one full training pass over a whole dataset of refinement states; WER runs multiple phases, each seeded by the previous phase’s retained mixed-outcome records.
• Round — one application of the optimizer’s operator to a single task (write one revision); a phase advances every task by one round.
• Mixed outcome / retention — the paper’s rule for keeping a candidate’s evidence for the next phase: keep it only if some of its rollouts succeeded and some failed (with n=2, exactly a 1/2 score), because that’s the case with both a working path and a failing path to compare.
• Pass@1 — the fraction of tasks solved correctly on a single attempt (as opposed to Pass@k, which allows k tries and counts a success if any of them work).
• BFCL (Berkeley Function-Calling Leaderboard) v4 multi-turn — a benchmark of multi-step, tool-calling tasks (File System, Trading, Travel, Vehicle domains) scored by comparing final environment state to a reference.
• τ²-bench — a benchmark of conversational tool-agent tasks (Airline, Retail, Telecom) with a stateful environment, an LLM-simulated user, and outcome-based scoring.