Training-Free Group Relative Policy Optimization

TL;DR

Agentic RL (GRPO and friends) makes LLM agents better at a domain by nudging their weights — which costs thousands of dollars, mountains of data, and a dedicated GPU deployment, and the result often overfits and generalizes poorly. This paper asks: what if we keep the model frozen and instead optimize the context? Training-Free GRPO runs the exact GRPO skeleton — sample a group of answers per question, compare them within the group — but replaces the numerical advantage (a gradient signal) with a semantic advantage: the model reads its own winning and losing rollouts and distills a short lesson in plain English. Those lessons accumulate in an “experience library” that gets prepended to future prompts. With ~100 training examples and roughly $8–$18 of API spend, a frozen DeepSeek-V3.1 beats 32B models that were fine-tuned with thousands of examples and >$10,000 of compute. The headline: on AIME math, +2.7% / +5.4% over an already-strong ReAct baseline; on WebWalkerQA web search, +4.6%.

Problem & Motivation

You run an AI services company. A client wants an agent that’s genuinely good at, say, their internal-document QA or a niche math-heavy workflow. The textbook answer in 2024–25 is agentic RL: collect domain data, do supervised fine-tuning, then a GRPO pass to sharpen behavior. That path has four concrete pains the paper lays out:

• Cost. Fine-tuning even a 32B model is expensive; tuning a frontier model is prohibitive. And once tuned, you must host that custom model — a standing GPU bill for what might be a low-frequency use case.
• Poor generalization. A model tuned for math gets worse at other things. So you end up maintaining a zoo of specialist models, each a separate deployment.
• Data scarcity. Real specialized domains rarely have thousands of clean, labeled examples. With few samples, fine-tuning overfits.
• Diminishing returns / the cost-performance dilemma. Budget forces you to fine-tune a small model, but a big general API model is often cheaper-per-quality and keeps improving for free. The small tuned model wins on the narrow task and loses everywhere else.

The core question: is parameter-space RL the only way to adapt an agent? The paper’s bet is that a frontier model already knows how to do the task — it just needs a little structured practice, and that practice can live in the prompt as a “token prior” rather than in the weights.

What’s New (Core Contribution)

• A training-free RL paradigm. Before: policy optimization = gradient ascent on model parameters. Now: policy optimization = editing a natural-language experience library that conditions the frozen model. Same GRPO loop structure, zero gradients.
• Semantic group advantage. Before: GRPO computes a numerical advantage Â_i = (r_i − mean(r)) / std(r) per rollout and uses it as a gradient signal. Now: the model reads the group of rollouts (winners and losers) and writes a plain-English lesson — a “semantic advantage” — that plays the same role: it tells the policy what to do more of. This is the genuinely new idea.
• Data + compute efficiency, demonstrated. ~100 examples, $8–$18, 3 epochs, no GPUs of your own — and it beats fine-tuned 32B specialists.
• Generalization preserved. Because weights never change, the base model stays general; you swap experience libraries the way you’d swap a system prompt. No model zoo.

Be honest about the framing: at its mechanical core this is agentic memory / self-reflection (à la Reflexion, Generative Agents) dressed in GRPO’s clothing. The novelty isn’t “LLMs can learn from written feedback” — it’s the disciplined GRPO-shaped loop (grouped rollouts, within-group comparison, batched Add/Delete/Modify/Keep edits, multi-epoch) applied to a frozen frontier model, plus the empirical receipts showing it rivals real RL at 1/1000th the cost.

How It Works (Technically)

Let’s first ground the RL vocabulary, because the whole paper is an analogy to it.

GRPO in one paragraph (the thing being mimicked). In RL for LLMs, the “policy” is the model itself: given a query q, it produces an output o with probability π_θ(o | q), where θ are the weights. GRPO (Group Relative Policy Optimization, from DeepSeek) skips the separate value network that PPO uses. For each query it samples a group of G answers, scores each with a reward r_i (e.g., 1 if the math answer is correct, else 0), and computes a group-relative advantage:

Â_i = (r_i − mean(r)) / std(r)

Read this as: how much better than its peers is answer i? Subtract the group mean (so “good” is relative to this batch), divide by the spread (so the signal is normalized). Answers above the mean get a positive advantage and their tokens get reinforced; below-mean answers get suppressed. A KL-divergence penalty against the original model keeps the policy from drifting too far (stability). Then gradient ascent nudges θ.

The substitution Training-Free GRPO makes. Keep θ frozen forever. Introduce an external experience library E (a list of short text lessons), initialized empty. The policy is now π_θ(o | q, E) — same frozen model, but conditioned on E shoved into the prompt. Optimization no longer touches θ; it edits E. Here’s the loop, step by step:

1. Rollout & reward. For query q, sample G outputs (G=5 for math, 3 for web) conditioned on the current E. Score each with a reward R(q, o_i) — for math, correctness against ground truth.
2. Filter to informative groups. Just like Â_i = 0 when all rewards are equal in GRPO (std = 0, nothing to learn), here a group is only useful if it has both a clear winner and a clear loser. Skip the all-right or all-wrong groups.
3. Compute the semantic advantage. First, the model summarizes each rollout: s_i = M(p_summary, q, o_i). Then, given all summaries and the current E, the model writes the reason the winners beat the losers and distills a concise lesson: A_text = M(p_extract, q, {s_i}, E). This A_text is the advantage — natural language instead of a number.
4. Update the library (the “gradient step”). Collect every A_text from the batch and ask the model to emit a list of edit operations on E:
   • Add — append a new lesson.
   • Delete — drop a lesson that’s proven low-quality.
   • Modify — refine an existing lesson with the new insight.
   • Keep — leave E unchanged.
5. Repeat for multiple epochs. Because E grew, the next epoch’s π_θ(o | q, E) produces a shifted output distribution — exactly the effect of a weight update, but achieved through context.

The clever framing: the frozen base model is itself the KL constraint. In GRPO the KL penalty stops the policy from drifting into incoherence; here, since weights never move, outputs are always anchored to the base model’s competence. You literally cannot blow up the policy, because you never touched it.

A concrete trace (math): q = an AIME problem. Sample 5 ReAct trajectories with a code interpreter. Two get the right answer using a clean closed-form; three loop on brute-force code and time out. The model summarizes all five, notices the winners avoided redundant tool calls, and writes A_text ≈ “For combinatorics problems, derive the formula symbolically before writing verification code; avoid brute-forcing the full search space.” The batch update Adds that lesson to E. Next epoch, every rollout sees it — and indeed the paper reports the average number of tool calls drops over training: the agent learned to be judicious, not just more accurate.

Architecture & data flow

flowchart TD
  Q[Training query q] --> R[Sample G rollouts<br/>conditioned on E]
  E[(Experience library E)] --> R
  R --> SC[Score each with reward R]
  SC --> F{Group has clear<br/>winner AND loser?}
  F -- no --> SKIP[Skip group]
  F -- yes --> SUM[Summarize each rollout: s_i]
  SUM --> EXT[Extract lesson A_text<br/>= semantic advantage]
  E --> EXT
  EXT --> OP[LLM emits edits:<br/>Add / Delete / Modify / Keep]
  OP --> E
  E -. prepended at inference .-> INF[Frozen model answers<br/>new domain queries]

Numerical vs. semantic advantage on one group of rollouts. Toggle between GRPO (computes a normalized number per answer to push gradients) and Training-Free GRPO (reads winners vs. losers and writes a text lesson). Same comparison, different output type.

The experience library evolving over 3 epochs. Watch lessons get Added, Modified, and Deleted as batches of semantic advantages arrive — this is the "policy update" without any gradient.

The algorithm, simplified

# Training-Free GRPO: optimize an experience library, not weights.
# llm(prompt) -> str ; reward(q, o) -> float ; model M is the SAME frozen llm.

def train_free_grpo(train_queries, epochs=3, G=5):
    E = []                                  # experience library (list of text lessons)
    for _ in range(epochs):
        batch_advantages = []
        for q in train_queries:
            # 1. rollout: G answers conditioned on the current library
            rollouts = [llm(prompt(q, E)) for _ in range(G)]
            rewards  = [reward(q, o) for o in rollouts]

            # 2. only informative groups teach anything (mirrors std(r)==0 -> Â=0)
            if max(rewards) == min(rewards):
                continue

            # 3. semantic advantage: read winners vs losers, distill a lesson
            summaries = [llm(p_summary(q, o)) for o in rollouts]
            A_text = llm(p_extract(q, summaries, rewards, E))   # natural-language "advantage"
            batch_advantages.append(A_text)

        # 4. "gradient step": one batched edit of the library from all advantages
        ops = llm(p_update(E, batch_advantages))   # -> [Add/Delete/Modify/Keep, ...]
        E = apply(ops, E)
    return E                                  # ship E as a system-prompt block

# inference: frozen model + learned library, on brand-new queries
def answer(q, E):
    return llm(prompt(q, E))

The whole “training” is apply(ops, E) — a text edit. There is no optimizer, no backprop, no GPU.

Built on Prior Work

Results & Evidence

Math (AIME24/25, Mean@32 = avg pass@1 over 32 runs), frozen DeepSeek-V3.1-Terminus:

• ReAct baseline: 80.0 / 67.9. With Training-Free GRPO: 82.7 / 73.3 (+2.7 / +5.4), ~$18, 100 examples.
• It beats RL-fine-tuned Qwen2.5-32B specialists (ReTool 67.0/49.3, AFM 66.7/59.8) that cost ~$10k–$20k.
• Works on smaller models too: Qwen3-32B +4.4/+5.9, Qwen2.5-72B +1.4/+1.8.

Web search (WebWalkerQA, pass@1): 63.2 → 67.8 (+4.6) on DeepSeek-V3.1.

Ablations — these are the load-bearing evidence:

• Directly generated experiences (ask the model to just write tips, same count, no loop): 80.0 → 79.8 — no gain. The GRPO-shaped loop, not just “having tips,” is what works.
• Group size = 1 (remove within-group comparison): drops to 80.4/69.3 vs 82.7/73.3. Confirms the relative comparison matters.
• No ground truth (advantage from majority-vote/self-consistency only): still 80.7/68.9 — usable where labels are scarce, though weaker than with labels.

Caveats you should weigh before selling this:

• Cross-domain transfer can hurt (Table 6). Experiences learned on Web lowered AIME scores; math-learned experiences hurt web. The “preserves generalization” claim is about the base model staying frozen — the library itself is a specialist and can be actively harmful out of its domain. So E is a per-domain artifact, like a system prompt, not a free lunch.
• Tested on exactly two task types, both with cleanish reward signals (math correctness, QA matching). No long-horizon, sparse-reward, or fuzzy-reward domains shown.
• It rides on a frontier model. The whole premise is “the model already knows how.” On a genuinely weak base model, there’s nothing to elicit. Gains on Qwen2.5-72B were small (+1.4/+1.8).
• Inference cost moves, not disappears. A growing E inflates every prompt’s token count at serve time — you trade a one-time training bill for a recurring context-length tax.
• No reported variance/CIs on the library-learning process; LLM-as-editor is stochastic and the paper doesn’t quantify run-to-run stability of E.

How You’d Use It

This is unusually practical for an AI services shop, because it needs no training infrastructure — just API calls and an eval harness.

• Client agent adaptation without a GPU bill. A client has a niche workflow and ~50–200 labeled examples. Run Training-Free GRPO over a weekend, ship the resulting E as a system-prompt block bolted onto your existing ReAct/tool agent. No custom model to host, no MLOps. This is a productizable offering: “domain adaptation in days, billed as a fixed-fee engagement.”
• Per-client experience libraries. Because E is just text, you can maintain a library per client/per-domain in version control, diff it, A/B test versions, and roll back. It’s a config artifact, not a model weight blob.
• MAS upgrade. In your multi-agent system, give each role its own learned library distilled from that role’s past successful/failed trajectories. The semantic-advantage loop is a principled way to turn agent run logs into role-specific operating procedures — far cleaner than ad-hoc prompt tinkering.
• Continual improvement loop. Pipe production traces (with outcome labels, even noisy ones) back through the loop periodically. The no-ground-truth variant means you can improve from self-consistency alone where you lack labels.
• Honest scoping: keep E domain-scoped (route to the right library per task), budget for the prompt-length cost at scale, and don’t promise gains on weak base models.

Build Your Own (Minimal Recipe)

You can build a working version in a day. The pieces:

1. A base agent — your existing ReAct loop (model + tools). Don’t change it.
2. A reward function — for a labeled task, exact-match/grader; cost: trivial. This is the part that decides whether the whole thing works, so make it tight.
3. The rollout-and-group step — call the agent G times per query (G=3–5), collect outputs + rewards. Use temperature ~0.7 for diversity during learning.
4. Three prompts — p_summary (compress one rollout), p_extract (read the group’s winners vs losers + current E, output one lesson), p_update (read all lessons + E, output Add/Delete/Modify/Keep edits as JSON). These three prompts are the project.
5. The library E — a plain list of strings; apply(ops, E) is a few lines.

Build order: reward function → single-query semantic-advantage extraction (verify the lesson is sensible by eye) → batched library update → multi-epoch loop → eval on held-out set.

The two genuinely hard parts: (a) p_extract quality — getting the model to write transferable, specific lessons rather than vague platitudes or problem-specific memorization; iterate on this prompt heavily. (b) Library hygiene — without disciplined Delete/Modify, E bloats into a contradictory mess; the ablation shows naive tip-dumping doesn’t help, so the editing is doing real work. Reach for: any frontier API model (the paper uses DeepSeek-V3.1), your existing agent framework, and a small eval set you trust.

How to Improve It

1. Retrieval-gated library. Instead of prepending all of E, embed each lesson and retrieve only the top-k relevant to the current query. Fixes the prompt-bloat tax and likely fixes the cross-domain-harm problem (Table 6) by never injecting off-domain lessons.
2. Lesson-level credit assignment. Tag which lessons were “active” when a rollout won or lost, then up/down-weight or prune at the lesson level — a finer-grained “gradient” than batch-level edits.
3. Learn the reward, too. Where ground truth is missing, train a lightweight LLM-judge reward alongside, rather than relying on majority vote; could close the gap to the labeled version.
4. Hierarchical libraries for MAS. Shared global lessons + per-role private lessons, with a promotion mechanism (a lesson that helps many roles graduates to global). Natural fit for your multi-agent work.
5. Stability + variance study. Run the learning loop N times, measure how much E and final scores vary; add a consistency check (only Add a lesson confirmed across multiple batches) to harden it for production.
6. Curriculum / harder-example mining. Spend the limited budget on the most informative (high-disagreement) groups first, since uniform-reward groups teach nothing anyway.

Glossary

• Policy (π_θ) — the model viewed as a function from query to output; θ are its weights. Here θ stays frozen and the prompt (E) does the adapting.
• GRPO — Group Relative Policy Optimization: an RL method that scores a group of sampled answers and reinforces the above-average ones; no separate value network.
• Advantage (Â_i) — how much better answer i is than its group peers; the signal RL uses to push the policy. This paper turns it into text.
• Semantic advantage (A_text) — the paper’s replacement for the numerical advantage: a plain-English lesson distilled from comparing winning vs. losing rollouts.
• Rollout — one full sampled trajectory of the agent answering a query (including any tool calls).
• Reward (R) — scalar quality score for a rollout (e.g., 1 = correct).
• KL-divergence penalty — in GRPO, a term that keeps the tuned policy close to the original to avoid degeneration; here the frozen base model provides this stability for free.
• Token prior / in-context learning — steering a model’s output by what’s in its prompt rather than by changing its weights.
• Experience library (E) — the accumulated, editable set of natural-language lessons that gets prepended at inference; the thing being “trained.”
• ReAct — an agent loop that interleaves reasoning steps and tool actions; used here as the base agent.
• Mean@32 / pass@1 — eval metrics: pass@1 = fraction correct on a single attempt; Mean@32 = that averaged over 32 independent runs for stability.
• Out-of-domain — evaluated on data distinct from the (small) training set; here training is DAPO-Math problems, eval is AIME.