KARL: Knowledge Agents via Reinforcement Learning

TL;DR

Enterprises need agents that can search proprietary data and reason over what they find (“grounded reasoning”) — narrowing candidates to one entity, synthesizing a report from scattered findings, doing math over financial tables. Existing benchmarks only test narrow slices, and models tuned for one slice don’t transfer to others. KARL does three things together: (1) it bundles six different search regimes into one benchmark (KARLBench) so you can measure generalization; (2) it builds training data with an agentic synthesis pipeline — an agent that explores the corpus with vector search and writes grounded question/answer pairs, filtered by difficulty and quality; and (3) it post-trains with OAPL, a large-batch off-policy RL method that is far simpler and cheaper to run than online GRPO and still trains big mixture-of-experts models stably. Starting from GLM 4.5 Air (a modest open model), KARL becomes Pareto-optimal against Claude 4.6 and GPT 5.2 on cost/quality and latency/quality — matching the best closed model (Opus 4.6) with enough parallel test-time compute, while costing less per query than even its own base model.

Problem & Motivation

The concrete pain: a model good at one kind of enterprise search is useless at the others, and there was no way to even measure that gap. A bank wants numerical reasoning over 100-page 10-Ks. A hospital wants a coherent report stitched from dozens of biomedical abstracts. A PM wants every customer who raised a governance concern, pulled from messy internal notes. These are structurally different problems — “deep search” (find one hard-to-find entity) versus “wide search” (exhaustively gather many facts) — and an agent optimized for one offers “no guarantee of competence on the others.”

Three things made this hard before KARL:

• Benchmarks tested narrow slices. HotpotQA, BrowseComp, FinanceBench each capture one behavior. “Deep research” results on the public web don’t obviously transfer to closed proprietary corpora with no Google to lean on.
• Training data was the bottleneck. Good grounded-reasoning data must be diverse, grounded in real documents, and hard enough to teach something. You can’t get that from prompting alone or from static synthesis (conditioning a generator on a fixed document blob).
• The RL that produces these agents was expensive and fragile. Online RL (GRPO) for large mixture-of-experts (MoE) models needed a pile of stabilization hacks — clipped importance weighting, deleting stale data, “router replay” — just to not diverge, because the model generating rollouts (vLLM) drifts from the model being trained.

The economic stakes are real: grounded reasoning over proprietary data is where the enterprise money is, and frontier API calls for multi-step agentic search get expensive fast.

What’s New (Core Contribution)

Four contributions, each a “before → now”:

• KARLBench (measurement). Before: you tuned to one benchmark and hoped. Now: six regimes in one suite — constraint-driven entity search (BrowseComp-Plus), cross-document report synthesis (TREC-Biogen), tabular numerical reasoning (FinanceBench), exhaustive entity retrieval (QAMPARI), procedural reasoning over docs (FreshStack), and fact aggregation over internal notes (PMBench, their proprietary one). All restricted to a single vector-search tool so you measure retrieval+reasoning, not tool orchestration luck.
• Agentic synthesis (data). Before: generate Q/A by conditioning on a static set of documents. Now: an agent actively explores the corpus with vector search before proposing each grounded Q/A pair, then a second stage runs N solver attempts to estimate each question’s difficulty and throws away the too-easy and too-hard. As the trained agent improves, you bootstrap it to synthesize the next round of data — self-improving data quality.
• OAPL: iterative large-batch off-policy RL (the algorithm). Before: online GRPO with stabilization heuristics to handle trainer/inference drift on MoEs. Now: a regression objective that embraces off-policyness, so trainer-vs-vLLM discrepancy stops being a bug. No importance-weight clipping, no data deletion, no router replay. Generate one big offline batch, do many updates and hyperparameter sweeps on it, optionally iterate. It’s cheaper and the infra is dramatically simpler.
• Multi-task generalization + test-time compute. Before: multi-expert distillation (train experts, SFT them into one model — how DeepSeek-V3.2 and GLM-5 were built). Now: just add the two task losses together with balanced token counts; this beats distillation on out-of-distribution generalization. Then “parallel thinking” (N rollouts + an aggregator that can itself use tools) and value-guided search push quality further at inference time.

How It Works (Technically)

There are three machines here: a data factory (agentic synthesis), a trainer (OAPL), and an inference-time booster (parallel thinking / value-guided search). The agent itself is dead simple by design — one tool (vector search), a loop, and an in-context compression step when the history gets too long.

The agent harness

The agent only has vector_search(query) -> chunks. It loops: read trajectory so far → emit a search query (a tool call) → read retrieved chunks → repeat → emit final answer. When the running context exceeds a token threshold, a compression plugin fires: the agent is asked to summarize its own history into a shorter blob, then continues. Crucially, KARL does not use a separate pretrained summarizer — compression is trained end-to-end inside RL using the task’s outcome reward, so the model learns what to keep because keeping the right thing earns reward later.

The data factory (agentic synthesis)

Stage I — Question/Answer synthesis. Seed the synthesizer with a few example Q/A pairs and the corpus. It explores via vector search (up to ~50-60 steps), then proposes grounded Q/A pairs (question + nuggetized answer + citations). A deduplication agent (LMSys decontamination pipeline + a paraphrase judge like gpt-4o-mini) strips anything that overlaps the eval set, so there’s no test leakage.

Stage II — Solution synthesis + filtering. Run the solver agent N independent times on each synthetic question. Its empirical pass rate is the difficulty estimate. Drop questions solved on all attempts (no learning signal) or none (unsolvable, wrong reference, or beyond reach). Survivors go to a Quality Filter agent that checks for ambiguity or a wrong reference answer. What’s left — plus the rollouts — becomes RL training data.

flowchart LR
  C[Corpus + few-shot examples] --> S[Q/A Synthesizer Agent<br/>explores via vector search]
  S --> D[Dedup Agent<br/>remove eval-set overlap]
  D --> Q[Synthetic Q/A pairs]
  Q --> M[N Solver Agents<br/>independent attempts]
  M --> P[Pass-rate filter<br/>drop all-correct & all-wrong]
  P --> QF[Quality Filter Agent<br/>ambiguity / bad reference]
  QF --> T[OAPL RL training data]
  T -.bootstrap improved model.-> S

OAPL: the RL objective, demystified

This is the heart. Start from the standard KL-regularized RL goal — maximize reward but don’t drift too far from a reference model π_ref:

maximize over π: E[ r(x, y) − β · KL(π ‖ π_ref) ]

In plain English: get high reward r on prompt x with response y, but pay a penalty β for moving away from the reference policy π_ref (the base model or the previous checkpoint). This regularized objective has a known closed-form optimum:

π*(y|x) ∝ π_ref(y|x) · exp( r(x,y) / β )

Meaning: the ideal policy is just the reference model re-weighted by an exponential of reward — good responses get up-weighted, scaled by temperature β. Define the optimal value V*(x) = β · ln E[ exp(r/β) ] (a soft-max over rewards = “how good is this prompt’s best achievable outcome”). Rearrange and you get the magic identity:

β · ln( π*(y|x) / π_ref(y|x) ) = r(x,y) − V*(x)

The right side, r − V*, is the advantage: how much better this particular response is than the prompt’s expected value. So learning π* becomes a plain least-squares regression: make the log-ratio (scaled by β) match the advantage. Concretely, the loss over a group of G rollouts {y_i} per prompt:

minimize over π: Σ_x Σ_i [ β·ln( π(y_i|x) / π_ref(y_i|x) ) − ( r(x,y_i) − V̂*(x) ) ]²

where V̂*(x) = β·ln( (1/G) Σ_i exp(r(x,y_i)/β) ) is estimated from the group’s own rewards (no separate critic network needed).

Why this matters operationally — three big consequences:

1. It’s off-policy by construction. The rollouts {y_i} were generated by π_ref (often via vLLM), not by the live policy π being updated. A regression target doesn’t care that the data is stale — it just fits the log-ratio to the advantage. So the trainer/inference mismatch that destabilizes GRPO is designed away. No importance-weight clipping, no router replay, no data deletion.
2. It’s a one-shot regression, not an online loop. Generate a big batch once, then hammer it with many gradient updates and hyperparameter sweeps — you amortize the expensive rollout generation. Optionally iterate: replace π_ref with the new policy, regenerate, repeat (≤3 iterations here).
3. Two temperatures, not one. In practice they split β into β₁ (controls smoothness of the value estimate V̂*) and β₂ (controls KL strength in the loss), for extra control.

Multi-step credit assignment. A rollout y is many steps (queries, retrieved docs, compressions, final answer). When computing ln π(y|x), they mask out tokens the model didn’t generate — the prompt, retrieved chunks, tool outputs. For very long rollouts they split at compression points into segments (x, y) where x is the compressed history and y is the steps until the next compression; the whole rollout’s reward is assigned to every segment. This keeps GPU memory bounded and folds the compression step itself into RL.

The algorithm, simplified

# OAPL: one iteration of iterative large-batch off-policy RL
# llm/policy calls return token logprobs; reward() grades a rollout in [0,1] or {0,1}
def oapl_iteration(prompts, pi_ref, beta1, beta2, G=8):
    batch = []
    for x in prompts:
        rollouts = [sample_rollout(pi_ref, x) for _ in range(G)]   # OFF-POLICY: data from pi_ref (vLLM)
        rewards  = [reward(x, y) for y in rollouts]
        # difficulty filter: a prompt teaches nothing if every attempt agrees
        if all(r == max(rewards) for r in rewards):                # all-correct or all-wrong
            continue
        # group-estimated optimal value: soft-max over the group's rewards
        V_hat = beta1 * logmeanexp([r / beta1 for r in rewards])
        for y, r in zip(rollouts, rewards):
            batch.append((x, y, advantage := r - V_hat))           # target = optimal advantage

    pi = clone(pi_ref)                                             # the policy we train
    for _ in range(many_updates):                                  # amortize: reuse the offline batch
        for (x, y, adv) in minibatches(batch):
            # only count tokens the model generated (mask prompt, retrieved docs, tool output)
            logratio = beta2 * (logprob(pi, y, x) - logprob(pi_ref, y, x))   # masked sum
            loss = ((logratio - adv) ** 2).mean()                 # least-squares regression to advantage
            loss.backward(); step(pi)
    return pi   # next iteration can set pi_ref <- pi, regenerate data, repeat

That regression loss — fit β·log-ratio to reward − V̂* — is the entire trick. Everything else (the data factory, the harness, TTC) wraps around it.

Test-time compute boosters

• Parallel Thinking: run N independent rollouts on the same prompt, then feed their short final answers to the same model acting as an aggregator — which can itself call tools. On PMBench, 23.7% of the time the aggregator with 5 rollouts produces an answer better than any individual rollout (so it’s strictly more expressive than Best-of-N or majority vote). Cheap on latency because rollouts run concurrently and the aggregator only reads short answers.
• Value-Guided Search (VGS): train a small value model σ(V(x, y≤t)) = “probability this partial rollout ends correct,” using a token-level cross-entropy loss against the binary outcome reward (a 4B model suffices). At each step, generate k=2 candidate continuations, keep the highest-value one (breadth-first search); run N such searches and aggregate. Task-specific but powerful (KARL-BCP jumps from 59.6 → 70.4 on BrowseComp-Plus).

Interactive: how OAPL turns a group of rollout rewards into per-rollout advantages. Drag the temperature β to see how V̂* (the group's soft-value) and each rollout's regression target (reward − V̂*) shift. Schematic, built to teach the loss — not the paper's exact numbers.

Interactive: parallel-thinking test-time compute. Slide N to watch quality climb with diminishing returns as more concurrent rollouts feed the tool-using aggregator. Schematic of the cost/quality trade-off the paper exploits.

Built on Prior Work

Results & Evidence

The headline (Table 4, KARLBench total, In-Dist / OOD / Total):

• Base GLM 4.5 Air: 52.6 total. KARL (multi-task, 2 iterations, no TTC): 58.9 — +6.3 over its own base, and cheaper per query.
• With parallel thinking: KARL N=3 → 64.1, N=10 → 67.5, N=20 → 68.1.
• Claude Opus 4.6 (best closed model): 67.5 total. KARL at N=10 matches it — at ~33% lower cost and ~47% lower latency. GPT 5: 60.1; GPT 5.2: 52.8; Sonnet 4.6: 62.3.
• Single-task experts confirm the regimes are genuinely different: KARL-TREC hits 85.0 on TREC-Biogen but only 42.2 on BrowseComp-Plus; KARL-BCP hits 59.6 (→70.4 with VGS) on BrowseComp-Plus but doesn’t transfer. Neither transfers to the other — multi-task RL is what buys the breadth.
• Generalization is the real story: KARL was trained only on BrowseComp-Plus + TREC-Biogen, yet improves on all four held-out OOD tasks (FreshStack, FinanceBench, QAMPARI, PMBench) — 51.2 → 53.7 OOD, amplified to 62.7 at N=10.
• Cost: single-call KARL scores competitively at under $0.10/query — lowest cost of any model above 55 points.
• Efficiency from RL: BrowseComp-Plus median trajectory drops from 50 steps (Iter 1) to 20 (Iter 2) — the model learns to search more efficiently, which is why it gets cheaper as it gets better.

What the evidence does NOT establish (read this honestly):

• Single tool only. Every result is vector-search-only. Real enterprise agents need structured queries, code execution, multi-tool orchestration — untested here, and the authors flag it as future work.
• Closed-corpus only. They deliberately avoid live web search for controlled comparison; generalization to noisy live retrieval is unverified.
• PMBench is proprietary and the strongest “enterprise” signal — you can’t reproduce or audit it.
• OOD = 4 tasks. “Out of distribution” still means other search benchmarks, not a genuinely novel domain or tool.
• A learned failure mode: KARL shows more “Giving Up Early” behavior — the authors suspect it learned a spurious “short trace ⇒ correct” correlation. RL found a shortcut.
• OAPL’s core math leans on a concurrent paper (Ritter et al. 2026) not yet public at read time; the stability claims for large MoEs rest partly on that.

How You’d Use It

For an AI services company, KARL is a blueprint for building a defensible, cost-efficient enterprise search agent on a client’s proprietary corpus — and three of its pieces are individually adoptable without touching RL:

• Agentic data synthesis as a service. The Stage I/II pipeline (explore-then-propose Q/A, pass-rate difficulty filter, quality filter) is a clean recipe for manufacturing grounded eval and training sets from any client corpus. You can run this with API models today, no GPU training. It’s a sellable deliverable: “we’ll build you a benchmark + a labeled training set grounded in your own documents, with provable no-leakage decontamination.”
• Parallel-thinking aggregation in your existing agent. This is pure inference-time wiring you can add to any client agent right now: fire N rollouts concurrently, let a tool-using aggregator merge them. It beat Best-of-N here and the latency cost is hidden by concurrency. Highest ROI, lowest effort.
• The harness discipline. “Identical harness from data collection → training → eval → serving” (their aroll design with lifecycle plugins for compression/budgeting) is the single most reusable systems lesson. Distributional shift between your eval rig and prod is a silent killer of agent quality; copy the principle even if you never train.
• The strategic argument: you can take a mid-size open model, specialize it on a client’s domain, and beat frontier APIs on cost and quality for that domain. That’s the moat pitch — a fine-tuned vertical agent that’s 10x cheaper per query than calling Opus, owned by the client.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value, in build order:

1. Single-tool agent + compression (1-2 days). A loop over vector_search, an in-context “summarize your own history” step at a token threshold. Use any decent open or API model. This is your harness — make it identical for synth, eval, and serving.
2. Nugget-based eval harness (1-2 days). Convert reference answers to atomic nuggets; score = fraction of nuggets covered (LLM judge). This unifies report-style and entity-style tasks under one number. Without this you can’t measure generalization.
3. Agentic synthesis pipeline (the highest-leverage piece, ~1 week). Stage I synthesizer-with-search + dedup judge; Stage II: N solver attempts → pass-rate filter (drop all-right/all-wrong) → quality filter. Now you have grounded, difficulty-calibrated data with no leakage.
4. Parallel-thinking TTC (1 day). N concurrent rollouts → tool-using aggregator. You’ll see most of the quality lift here before you ever train.
5. Only if you have GPUs and a reason: OAPL (the hard part). Generate a big offline batch with π_ref, compute group V̂* and per-rollout advantages, regress β·log-ratio → advantage with proper token masking and compression-segment splitting. Reach for: a serving engine (vLLM) for rollout generation, a training stack that gives per-token logprobs, GLM 4.5 Air or similar as base.

The two genuinely hard parts: (a) the data factory’s filtering — getting difficulty estimation and the leakage-free dedup right is what makes or breaks training; (b) OAPL’s multi-step credit assignment — masking non-model tokens and segmenting long rollouts at compression points without corrupting the advantage signal. Steps 1-4 are buildable by a competent team in ~2 weeks and deliver real client value; step 5 is a research-grade investment.

How to Improve It

1. Expand the action space. The authors’ own #1 lever: add structured retrieval (SQL/filters), code execution, and callable sub-agents. Test whether OAPL still generalizes when the tool surface is wide — that’s the untested frontier for real enterprise use.
2. Attack the “Giving Up Early” shortcut. RL learned “short trace ⇒ correct.” Add a reward term that penalizes premature termination on questions whose answers required more evidence, or a verifier that gates early commits. Cheap to test, directly fixes a measured regression.
3. Hierarchical / external memory instead of flat compression. Current compression is a one-shot self-summarization that throws away detail. Try a structured scratchpad or retrieval-over-history so the agent can recover dropped facts instead of losing them — likely fixes the “Running Out of Context” failures.
4. Process rewards from the value model. They already train a token-level value model for VGS. Fold σ(V(x, y≤t)) back into OAPL as a dense per-step shaping signal, not just a test-time selector — could speed convergence and improve credit assignment on long rollouts.
5. Curriculum over the difficulty filter. They binary-filter all-right/all-wrong. Instead, schedule difficulty — start near the model’s pass-rate frontier and ratchet up across iterations — to extract more signal per rollout, especially on the deep-search (BrowseComp) tasks where Iter 1 maxed out the step budget.

Glossary

• Grounded reasoning — reasoning that requires retrieving knowledge outside the model’s weights (e.g., a client’s proprietary docs).
• Vector search — retrieval by embedding the query and finding nearest-neighbor document chunks in embedding space; KARL’s only tool.
• Off-policy RL — training on data generated by a different (older) policy than the one being updated; the opposite of online RL where you train on fresh self-generated data.
• GRPO — Group Relative Policy Optimization; popular online RL for LLMs that compares rollouts within a group; needs hacks to stay stable on large MoE models.
• OAPL — KARL’s method: Optimal Advantage-based Policy optimization with Lagged inference; a least-squares regression of the log policy-ratio onto the optimal advantage.
• Advantage — how much better a specific response is than the prompt’s expected value; here, reward − V̂*.
• V̂*(x) (optimal value estimate) — a soft-max (log-mean-exp) over a group’s rewards; “how good is the best achievable outcome on this prompt.”
• β (temperature / KL strength) — knob trading off reward-chasing vs. staying close to the reference model; KARL splits it into β₁ (value smoothness) and β₂ (KL strength).
• KL regularization — penalty for the new policy drifting from a reference policy, keeping training stable.
• MoE (Mixture of Experts) — a model where each token routes to a few specialized sub-networks; efficient but historically painful to RL-train stably.
• Nugget-based evaluation — scoring an answer by the fraction of atomic ground-truth facts (“nuggets”) it covers; works for both single-entity and report-style answers.
• Token masking (in the loss) — only counting log-probs of tokens the model actually generated, ignoring prompt/retrieved-doc/tool-output tokens.
• Compression — the agent summarizing its own trajectory in place when context grows too long; here trained end-to-end via RL reward.
• Parallel Thinking — test-time compute strategy: run N rollouts concurrently, then a tool-using aggregator merges their answers (more expressive than Best-of-N).
• Value-Guided Search (VGS) — step-level tree search steered by a small trained value model predicting eventual success from a partial rollout.
• In-distribution / out-of-distribution (here) — relative to KARL’s training tasks (BrowseComp-Plus, TREC-Biogen); the other four KARLBench tasks are held-out OOD.
• Pareto-optimal — no other model is both cheaper/faster and higher quality; KARL sits on the efficient frontier.
• Test-time compute (TTC) — spending more inference compute (parallel rollouts, search) to raise quality without retraining.