Escaping the Verifier: Learning to Reason via Demonstrations

TL;DR

Most of the recent leap in LLM reasoning (DeepSeek-R1 and friends) comes from reinforcement learning with verifiable rewards — you train on math or code where a script can mechanically check “correct / incorrect.” But the tasks businesses actually care about — analytical writing, research, financial analysis — have no such verifier. RARO (Relativistic Adversarial Reasoning Optimization) removes that requirement. It only needs a dataset of expert demonstrations (question → good answer). It sets up an adversarial game: a policy writes chain-of-thought answers, and a critic (the same model, sharing weights) is shown one expert answer and one policy answer and must pick which is the expert. The policy is rewarded for fooling the critic; the critic is rewarded for not being fooled. Both train jointly via RL. The headline result: on Countdown it nearly matches a true verifier (54.4% vs 57.7% oracle), it beats every verifier-free baseline on general math (DeepMath) and on open-ended Poetry Writing, and — crucially — it shows the same “more reasoning budget → better answers” scaling curve that makes RL-with-verifiers so powerful.

Problem & Motivation

The pain in one sentence: RL gives LLMs real reasoning, but RL needs a reward signal, and outside math/code you don’t have one.

Walk through why the obvious alternatives fall short:

• SFT (supervised fine-tuning on demonstrations). Just copy the expert’s answers via next-token prediction. Two problems. First, it teaches the model to output good answers, not to reason toward them — you don’t get the explore-verify-backtrack behavior that makes reasoning models strong. Second, there’s a train/inference mismatch: during SFT the model always sees the real expert context, but at inference it conditions on its own previously-generated tokens. It never practices recovering from its own mistakes.
• RLHF / DPO (preference learning). Works, but needs preference data (“answer A is better than B”), which is expensive human labeling on top of the demonstrations you already have.
• Logit-based rewards (RL-Logit). Recent trick: reward the policy by how high a probability it assigns to the expert answer. Clever, but the reward is only as good as the base model’s own probabilities — and for hard tasks those are noisy/uninformative. In the paper it barely beats the base model and sometimes collapses.

Meanwhile, expert demonstrations are everywhere and underused: highly-upvoted Stack Exchange answers, published financial analyses, edited essays, curated poetry. The question RARO answers: can we extract reasoning-grade training signal from demonstrations alone, with no verifier and no preference labels?

What’s New (Core Contribution)

1. Reasoning-as-Inverse-RL framing. Before: reasoning RL meant “have a verifier, maximize verifiable reward.” Now: treat it as Inverse Reinforcement Learning — learn the reward function under which the expert demonstrations look near-optimal, then optimize the policy against that learned reward. This sidesteps an intractable computation (more below) and reframes the whole problem.
2. The critic IS a reasoning model, not a scalar head. Before: a reward model is usually a frozen classifier or a small value head. Now: the critic is a full LLM that reasons about which answer is the expert. They argue the judge must be at least as smart as the thing it judges. Critically, the critic shares weights with the policy — one model wearing two hats.
3. Relativistic (pairwise) critic with a “tie” option. Before: a naive binary “is this answer expert or policy?” classifier. Now: show the critic both answers and ask “which is better, or tie?” This fixes a fatal degeneracy: when the policy gets as good as the expert, a binary classifier collapses to random guessing and emits garbage gradients. The pairwise+tie setup gives a stable signal even at equilibrium. (This is the load-bearing trick — without it training is unstable.)
4. A recipe of stabilizers that make adversarial RL actually converge: shared weights + data-mixing of policy/critic rollouts in one batch, a replay buffer so the critic doesn’t forget old policy answers (catastrophic forgetting), and GRPO tweaks (over-length filtering, removing advantage/length normalization).

The honest read: IRL-as-adversarial-imitation is not new (GAIL did it in 2016). The genuine novelty is making it work for reasoning LLMs — the reasoning critic, the relativistic+tie formulation, and the stabilization recipe that keeps a jointly-trained adversarial pair from collapsing.

How It Works (Technically)

The core idea, demystified

You want a policy π(a, z | q) that, given a question q, generates a chain-of-thought z and an answer a that look like the expert’s. The “honest” objective is maximum likelihood: make the expert’s answers as probable as possible under your model.

The catch: a reasoning model produces an answer a via some hidden reasoning trace z. To compute the probability of just the answer a, you’d have to sum over every possible reasoning trace that leads to it — an astronomically large set. So π(a|q) = Σ_z π(a,z|q) is intractable. You can’t directly do maximum likelihood.

Inverse RL is the escape hatch. Instead of maximizing likelihood directly, learn a reward r_φ(a,q) such that the policy that optimizes that reward happens to also maximize likelihood. There’s a known result (KL-regularized reward maximization) that the optimal policy under a reward has a clean closed form:

π*(a|q) ∝ π_ref(a|q) · exp( r_φ(a,q) / β )

In plain English: the best policy is the reference model, tilted toward answers the reward likes, with β controlling how far you’re allowed to drift from the reference. Small β = chase the reward hard; large β = stay close to the original model. This is the exact same formula that underlies DPO/RLHF — worth recognizing.

Because that form is closed, you can take the gradient of the likelihood with respect to the reward parameters φ, and it simplifies to something beautifully intuitive:

∇_φ L ∝ E[ ∇_φ r_φ(expert answers) ] − E[ ∇_φ r_φ(policy answers) ]

Read this as: “push the reward UP on expert answers, push it DOWN on the policy’s own answers.” That is literally the discriminator objective from GANs. The reward model learns to score experts high and the current policy low; the policy then chases that reward, getting more expert-like; the reward must adapt; repeat.

From reward to a critic

Rather than a scalar reward head, they make r_φ a reasoning critic c_φ. Two formulations:

v1 (binary). Critic sees (q, a) and classifies expert vs policy. The reward r_φ is the probability it says “expert.” Plug into the gradient above and you get clean policy-gradient rewards:

• Critic reward = 1 if it classified correctly.
• Policy reward = 1 if the critic was fooled into saying “expert.”

This is a clean adversarial game — but it has the degeneracy problem. As the policy nears the expert, “is this expert or policy?” with no reference becomes a coin flip. The critic degenerates to 50/50, its gradients are pure noise, and training destabilizes.

v2 (relativistic — the actual method). Critic sees a triplet (q, a⁽¹⁾, a⁽²⁾) — one expert answer and one policy answer, shuffled — and returns {1, 2, tie}. Now the rewards carry an explicit tie term:

• R_critic = 1[picked the expert] + τ_crit · 1[said tie]
• R_policy = 1[critic picked the policy as expert] + τ_pol · 1[said tie]

The τ tie-rewards (both in [0,1]) are new hyperparameters. The tie option is the key stabilizer: when the policy genuinely matches the expert, the critic can honestly say “tie” and both sides get a stable, non-degenerate reward instead of fighting over a coin flip. The paper shows ablating “tie” hurts a lot.

Architecture & data flow

flowchart TD
  Q[Question q from expert dataset] --> POL[Policy pi_theta]
  Q --> EXP[Expert answer a_E from dataset]
  POL -->|CoT z + answer a_P| PAIR[Build pair: expert a_E + policy a_P, shuffled]
  EXP --> PAIR
  PAIR --> CRITIC[Relativistic Critic c_theta: which is expert? or tie]
  CRITIC -->|fooled? -> reward policy| RPOL[R_policy = fooled + tie bonus]
  CRITIC -->|correct? -> reward critic| RCRIT[R_critic = correct + tie bonus]
  RPOL --> GRPO[Joint GRPO update on shared theta]
  RCRIT --> GRPO
  GRPO -->|same weights| POL
  GRPO -->|same weights| CRITIC
  PAIR -.store.-> RB[(Replay buffer R)]
  RB -.sample old pairs.-> CRITIC
  style CRITIC fill:#e8f0ff
  style GRPO fill:#fff0e8

Note the self-play loop: one shared model θ is both the policy and the critic. A batch mixes “policy rollouts” (the model reasoning toward answers) and “critic rollouts” (the model judging answer pairs, including old pairs pulled from the replay buffer). One GRPO step updates everything.

Schematic of the RARO equilibrium: drag the slider to make the policy weaker or stronger. Watch the critic's accuracy and the "tie" rate respond — the tie option is what keeps the reward signal alive as the policy approaches the expert (the right edge), instead of collapsing to a noisy coin flip.

The algorithm, simplified

# RARO core loop. One shared model `theta` plays BOTH roles.
# llm_policy(q) -> (cot, answer) ; llm_critic(q, a1, a2) -> {1, 2, "tie"}
replay = []                                   # all past (q, expert, policy) triplets

for step in range(T):
    batch = sample_expert_QA(D, B)            # (q, expert_answer) pairs

    # --- 1. Policy rollouts: generate answers, score by fooling the critic ---
    new_triplets, policy_rewards = [], []
    for (q, a_E) in batch:
        for k in range(K):                    # K rollouts per question (GRPO group)
            cot, a_P = llm_policy(q)          # the model REASONS toward an answer
            label = llm_critic(q, *shuffle(a_E, a_P))      # who's the expert?
            r = 1.0*(label == "policy_is_expert) + TAU_POL*(label == "tie")
            policy_rewards.append(r)          # reward = critic got fooled (+tie bonus)
            new_triplets.append((q, a_E, a_P))

    # --- 2. Critic rollouts: judge fresh AND replayed pairs (anti-forgetting) ---
    critic_batch = mix(new_triplets, replay)  # data-mixing in a single batch
    replay += new_triplets
    critic_rewards = []
    for (q, a_E, a_P) in critic_batch:
        label = llm_critic(q, *shuffle(a_E, a_P))
        r = 1.0*(label == "expert") + TAU_CRIT*(label == "tie")
        critic_rewards.append(r)              # reward = correctly ID'd the expert

    # --- 3. One joint GRPO update on shared weights, with KL leash to ref model ---
    loss = (LAM_POL*grpo(policy_rewards) + LAM_CRIT*grpo(critic_rewards)
            - BETA*kl(theta, ref))            # advantage = reward - group mean
    theta = grpo_step(theta, loss)

The whole contribution lives in those reward lines: the policy’s reward is literally “did I fool the critic,” the critic’s is “did I catch the fake,” and GRPO turns each rollout’s reward into an advantage by subtracting the group mean. No verifier anywhere.

(Quick GRPO primer for the reader: GRPO = Group Relative Policy Optimization, DeepSeek’s PPO simplification. For each question you sample a group of K answers, compute each one’s reward, and define advantage = reward − mean(group rewards). Answers above the group average get reinforced, below-average get suppressed. No separate value network needed — the group average is your baseline.)

Built on Prior Work

Results & Evidence

Setup: Qwen2.5 instruct models (1.5B / 3B / 7B), 2048-token reasoning budget, three tasks chosen to span the verifiability spectrum.

Countdown (combine four numbers to make 24 — verification trivial, search hard). At 1.5B:

• RARO 54.4%, best verifier-free baseline (SFT) 40.7%, oracle RLVR 57.7%. RARO nearly matches the verifier it doesn’t have.
• RL-Logit (2.2%) and Rationalization (12.5%) basically fail.
• Scaling: RARO climbs 33.1% → 61.3% as reasoning budget goes 256 → 4096 tokens, and extrapolates (a 2048-trained model hits 61.3% at 4096 test-time tokens). SFT flatlines at 40.7% regardless of budget. This budget→accuracy curve is the real signal that genuine reasoning emerged.
• Qualitatively, the policy learns explore → verify → backtrack (“too high”, try again) — an internal verifier — which SFT never does.

DeepMath (general math, verification ≈ as hard as generation). RARO beats every verifier-free baseline and the gap grows with scale: +3.6% at 1.5B, +6.0% at 3B, +8.2% at 7B (57.5% vs RL-Logit 49.3%). Test-time scaling via a critic-judged single-elimination tournament: 1→16 rollouts pushes 1.5B to 53.6%, matching RLVR’s improvement rate.

Poetry Writing (no verifier exists; GPT-5 judges). The intended regime. RARO win-rate vs expert poems: 7.8% (1.5B) → 25.0% (7B), a 4× improvement over best baseline (5.9%). Scores 77.3 vs SFT 65.4 at 7B. RL-Logit, the DeepMath champ, collapses here (near-zero over base).

What the evidence does NOT establish — read this before you sell it:

• Small models only (≤7B). Authors flag scaling to SOTA sizes as future work. Joint adversarial RL at 70B+ is unproven and likely harder to stabilize.
• RLVR is still the ceiling where a verifier exists (RARO 54.4 vs 57.7 on Countdown). RARO is “best when you can’t verify,” not “better than verifying.”
• Poetry is judged by GPT-5, which is itself a (very strong) LLM judge — circular-ish, and win-rates are still modest (25%).
• Sample efficiency is a stated weakness (you’re running two RL games at once). No wall-clock/compute comparison vs SFT is foregrounded.
• Datasets are narrow (one custom poetry set, one math set, one toy task). Generalization to messy real domains (legal, financial) is inferred, not shown.

How You’d Use It

For an AI services company, this is the technique for the “we have great examples but no grader” client. That describes most knowledge-work automation: a law firm’s best memos, a consultancy’s past analyses, an agency’s winning ad copy, a clinic’s exemplar patient notes. You can’t write a verify() for “good memo,” and clients won’t pay to hand-label thousands of preference pairs — but they do have an archive of expert outputs.

Concrete slots:

• Domain reasoning models as a productized offering. Take a client’s demonstration corpus → fine-tune a small open model (Qwen/Llama 7B class) with RARO → ship a model that reasons in their house style and domain, not just parrots it. The differentiator vs SFT is the emergent planning/self-correction, which shows up as visibly better answers on novel inputs.
• Reusable critic as an eval/reranker. The trained critic is a pairwise “which is better” judge for that domain — drop it into a Best-of-N or tournament reranker at inference time (the paper’s TTS result), or use it as an automated regression check in a CI pipeline for content quality.
• Replace brittle LLM-as-judge rubrics. Instead of hand-crafting rubric prompts for GPT-4 to grade outputs, you get a learned judge calibrated to actual expert data.

Effort/payoff read: this is a training-time technique, so it’s heavier than prompt engineering — you need an RL stack (TRL/verl), GPUs, and a clean demonstration dataset (hundreds–thousands of QA pairs). But it’s far cheaper than collecting human preferences, and the moat is real: a critic+policy tuned on a client’s proprietary archive is genuinely hard for competitors to replicate.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

1. Pick a small instruct model (Qwen2.5-1.5B/3B-Instruct) and an RL library that already does GRPO (trl’s GRPOTrainer, or verl). Don’t write GRPO from scratch.
2. Prepare demonstrations: a dataset of (question, expert_answer) pairs. A few thousand is plenty for a toy.
3. Two prompt templates on one shared model:
   • Policy prompt: “Reason step by step, then answer: {q}.”
   • Critic prompt: “Here are two answers to {q}. Answer A: … Answer B: … Reason, then output A, B, or TIE for which is the expert.”
4. The loop (the pseudocode above): per step, roll out K policy answers per question, pair each with its expert answer (shuffle order!), have the critic judge, compute the two reward signals, mix in some replay-buffer pairs for the critic, and take one joint GRPO step with a KL penalty to a frozen reference.
5. Tune the few hyperparameters that matter: τ_pol, τ_crit (tie rewards — start ~0.5), β (KL strength), and λ_pol/λ_crit (loss weights to balance the two games).

The two genuinely hard parts:

• Stability. Adversarial RL wants to collapse (mode collapse, critic forgetting, reward hacking). The tie option, the replay buffer, and shared weights are not optional polish — they’re what makes it converge. Watch reward + response length: healthy training shows both growing smoothly (their Fig. 3), not spiking/crashing.
• Avoiding shortcut critics. The critic can cheat on surface features (length, formatting, a telltale phrasing). Shuffle answer order, normalize formatting, and sanity-check that the critic is judging content, or your policy will learn to game artifacts instead of reasoning.

How to Improve It

1. Decouple the critic for a few steps when it collapses. Shared weights save memory but couple failure modes. Add an adaptive schedule: if critic accuracy drifts to ~50% (mode collapse), give the critic extra solo rollouts before resuming joint updates. Testable against their DeepMath instability ablation.
2. Curriculum on critic difficulty. Start the critic on easy pairs (base-model vs expert) and harden as the policy improves, instead of always pairing current-policy vs expert. Could improve sample efficiency (their stated weakness).
3. Calibrated/soft critic instead of hard {1,2,tie}. Have the critic emit a probability (or a margin), and use it as a graded reward. Denser signal → lower-variance advantages → faster convergence. The tie option is a coarse 3-bucket version of this.
4. Verifier-when-available hybrid. Where a partial verifier exists (e.g., math answer-checking but not proof-quality), blend R = α·verifier + (1−α)·critic. RARO already nearly matches RLVR; a hybrid might beat pure RLVR by adding style/quality signal the verifier ignores.
5. Critic ensemble / cross-critic to fight reward hacking. Train two critics with different prompts/seeds; the policy must fool both. Harder to game surface artifacts; standard GAN-stabilization wisdom that the paper doesn’t try.
6. Interpretable critic. The authors list “reward interpretability” as future work. Make the critic emit a short natural-language rationale for its choice — turning the reward into verbal feedback the policy can condition on (Reflexion-style), bridging adversarial RL and self-improvement.

Glossary

• RARO — Relativistic Adversarial Reasoning Optimization; this paper’s method.
• Verifier / RLVR — a mechanical correctness checker; RL with Verifiable Rewards uses it as the reward. Available for math/code, absent for writing/analysis.
• Demonstrations — expert (question, answer) examples, with no quality labels or preferences attached.
• Inverse Reinforcement Learning (IRL) — instead of being given a reward, you infer the reward function that makes observed expert behavior look optimal.
• Policy (π) — the model being trained to produce answers; here it also produces chain-of-thought.
• Critic / discriminator — the model that judges whether an answer came from the expert or the policy; provides the learned reward.
• Relativistic critic — a critic shown both answers at once and asked which is better (pairwise), rather than scoring one in isolation.
• Tie option — letting the critic say “these are equal,” which keeps the reward signal stable when the policy matches the expert.
• GRPO — Group Relative Policy Optimization; RL update where each answer’s advantage is its reward minus the average reward of its sampled group (no value network needed).
• Advantage — how much better an action’s reward is than a baseline; positive → reinforce, negative → suppress.
• KL regularization (β) — a penalty keeping the trained policy close to a frozen reference model, preventing reward hacking/drift.
• GAIL — Generative Adversarial Imitation Learning; the 2016 ancestor that framed imitation as a policy-vs-discriminator game.
• Replay buffer — a store of past (expert, policy) pairs replayed to the critic so it doesn’t forget how to spot old policy mistakes (catastrophic forgetting).
• Catastrophic forgetting — when a model loses earlier-learned ability while learning something new.
• SFT — Supervised Fine-Tuning; plain next-token training on demonstrations.
• Test-Time Scaling (TTS) — spending more inference compute (more rollouts) to get better answers; here, a critic-judged tournament over candidate answers.