WebGPT: Browser-assisted question-answering with human feedback

TL;DR

Long-form question answering (write a paragraph, not a single fact) was stuck because retrieval and synthesis were trained separately and there was no good way to check if the model’s paragraph was actually true. WebGPT fixes both problems with one trick: put GPT-3 inside a text-based browser (search, click, scroll, quote) so it collects its own evidence, then have humans compare pairs of full “browsing session + answer” transcripts and train a reward model on those preferences. The policy is optimized against that reward model two ways — reinforcement learning (PPO) and, more effectively, just sampling many answers and keeping the best-scored one (rejection sampling / best-of-n). The result: a 175B model that beats its own human demonstrators 56% of the time and beats the top-voted Reddit answer 69% of the time on ELI5, and is meaningfully more truthful than raw GPT-3 on TruthfulQA. It’s also the direct ancestor of ChatGPT’s “Browse with Bing” / browsing tool-use and a template for citation-grounded RAG products.

Problem & Motivation

Long-form QA (LFQA) means: user asks an open-ended question, model writes a paragraph-length answer, not a single span or entity. Two things made this hard before WebGPT:

• Retrieval and synthesis were bolted together, not trained together. Systems like REALM and RAG train a differentiable retriever (dot-product search over document embeddings) jointly with a generator — but that only works for retrieval you can express as a similarity search. It can’t call a real search engine, click through search-result pages, or decide to search again after reading something. Krishna et al. (2021), the previous best system on ELI5, used this style of retriever and their answers were preferred to Reddit’s top answer only 23% of the time.
• There was no reliable way to know if an answer was true. Automated metrics like ROUGE-L barely correlate with quality for open-ended generation — Krishna et al. found this directly. And asking a human labeler “is this factually accurate?” with no evidence to check against forces them into slow, subjective, expert-level fact-checking for every claim.

If you can’t score truthfulness, you can’t optimize for it. That’s the actual bottleneck this paper attacks — not “can a language model write good prose” (GPT-3 could already do that), but “how do you get reliable enough feedback on a long, unverifiable answer to train against it.”

What’s New (Core Contribution)

1. A real, non-differentiable web browser as the retrieval mechanism, controlled by discrete text commands. Before: retrieval was a differentiable inner-product search over a fixed corpus. Now: the model emits actual actions — Search, Click, Find in page, Scroll, Back, Quote — against live Bing search results and real web pages, the same way a human would. This is trained with plain policy-gradient-style methods (behavior cloning + RL + rejection sampling), not backprop-through-retrieval.
2. Answers ship with references, and references are the unit of feedback. Before: labelers judged raw claims for truthfulness (hard, slow, subjective). Now: every claim the model wants credit for must come from a quote the model itself collected while browsing; labelers only judge “is this claim supported by this specific source,” which is a much easier, more consistent task (researcher–labeler agreement rose to 74%). This is the paper’s most exportable idea — it’s the design pattern behind every “answer with citations” product since.
3. Rejection sampling (best-of-n) as a first-class optimization method, not just an eval trick. Before: RLHF papers (Stiennon et al. 2020, which this paper directly extends) used RL as the optimization method against a reward model. WebGPT shows that for this task, simply sampling n answers from the behavior-cloned policy and keeping the reward-model’s favorite beats RL outright (68% preferred over BC vs. RL’s 58%), while needing zero extra training — you trade inference-time compute for quality instead of training-time compute.
4. A stateless-per-step environment design that sidesteps long-context RL. The model has no memory across browsing steps except a hand-written textual summary of past actions re-inserted into the prompt every turn. This keeps each decision a short, well-scoped completion instead of requiring the model to reason over an ever-growing transcript — a deliberate simplicity trade against “give the model perfect memory.”

How It Works (Technically)

The environment. The model doesn’t see raw HTML. Every turn it gets a text rendering of “the current webpage state” (question, page title, the visible slice of page text at the current scroll position, the list of past actions it has taken, and how many actions it has left) and must output exactly one command from a fixed list:

Links are rewritten into a compact token format (【<id>†<link text>†<domain>】) and Reddit/Quora are blacklisted from results so the model can’t just copy the answer it’s being trained to reproduce. Pages that overlap 10 grams with the question are censored the same way, to stop it from finding a “answer key.” Browsing ends when the model calls End: Answer, hits the action budget (20–100 actions, randomized during RL), or runs out of reference-length budget. At that point — and only if it collected at least one quote — a separate prompt (question + collected quotes) is used to compose the final answer with inline citations [1] [2].

Why this design matters mechanically: because the browsing phase and the answering phase are two different completions, and the browsing phase has no persistent hidden state (memory is just re-serialized text), every “turn” is a short, ordinary language-model completion — Search, Click, Quote, etc. are unusual only in that they act on the world. This is exactly the ReAct-style “observe → think → act” loop, four years before that name existed, implemented with fine-tuning + RL instead of pure prompting.

Data collection. Two kinds of human-generated data:

• Demonstrations (~6,200): a human uses a GUI version of the same environment to fully browse and answer a question. This is the imitation-learning signal — it teaches the model the format of valid commands and reasonable browsing behavior, since a vanilla pretrained GPT-3 has no idea these commands exist.
• Comparisons (~21,500): two model-generated answers (with their own independent browsing sessions and references) to the same question are shown to a labeler, who rates which is better — using a 5-point Likert scale collapsed to “better/worse/tie” — after separately judging how well each claim is supported by references. This is the preference signal.

Four training methods, all standard in a specific sense — the paper doesn’t invent new algorithms, it applies known ones to a new action space:

1. Behavior cloning (BC) — plain supervised fine-tuning of GPT-3 on demonstrations, where “labels” are the commands a human issued. This is what teaches the model to use the browser at all.
2. Reward modeling (RM) — take the BC model, chop off its output (unembedding) layer, and train a scalar head to predict a reward r(question, answer+references) from the comparisons, via a Bradley–Terry / Elo-style loss: P(A preferred to B) = sigmoid(r(A) - r(B)) Trained with cross-entropy against the human “A better / B better / tie” labels (ties = soft 50% label). Plain English: the reward model learns a single number per (question, answer) such that the difference between two such numbers predicts how often humans will prefer one over the other — a 1-point gap means about 73% preference (sigmoid(1) ≈ 0.73).
3. Reinforcement learning (RL) — fine-tune the BC model with PPO, treating the whole episode (browsing + answer) as a trajectory. Reward = reward-model score at the very end, plus a per-token KL penalty against the frozen BC model (this stops the policy from drifting into gibberish that fools the reward model — classic “don’t overoptimize a proxy” regularization). To make this sample-efficient despite most of the reward-model signal coming from the answer text rather than the browsing actions, they append 15 extra answer-only episodes (reusing the same collected references) after every one browsing episode — roughly doubling effective sample efficiency for free.
4. Rejection sampling (best-of-n) — no training at all. Sample n independent full episodes (n = 4, 16, or 64) from the BC (or RL) policy, score each with the reward model, keep the single best-scoring one. This is literally “run inference n times, pick the winner” — the entire method is n and a max().

The winning recipe: BC, then best-of-n on top of BC (RL is dropped from the final models — it helps a little on its own, but adds nothing once you’re already doing rejection sampling, and costs a lot of extra training + hyperparameter tuning).

Architecture & data flow

flowchart TD
  Q[Question] --> POL["Policy (fine-tuned GPT-3)"]
  POL -->|emits one command per turn| CMD[Search / Click / Scroll / Find / Quote]
  CMD --> ENV["Text browser env\n(Bing API + page fetch + Readability.js)"]
  ENV -->|new page text, action budget left, quotes so far| SUMMARY[Re-serialized state + action history]
  SUMMARY --> POL
  POL -->|End: Answer, or budget exhausted| STOP{Has ≥1 quote?}
  STOP -->|yes| ANS["Answering-phase prompt\n(question + collected quotes)"]
  ANS --> OUT["Final answer with [1][2] citations"]
  STOP -->|no| DROP[Episode discarded / no answer]

flowchart LR
  DEMO[Demonstrations\n~6,200 human sessions] --> BC["Behavior Cloning\n(supervised fine-tune GPT-3)"]
  COMP[Comparisons\n~21,500 human prefs] --> RM["Reward Model\n(BC + scalar head)"]
  BC --> RL["RL (PPO) vs RM"]
  BC --> BON["Rejection sampling\nsample n, keep argmax RM(answer)"]
  RM --> RL
  RM --> BON
  RL --> BON2["RL + rejection sampling\n(marginal extra gain)"]
  BON --> WEBGPT["WebGPT\n(BC + best-of-n is the shipped model)"]

The per-turn environment loop. Each step shows one command the policy can issue, the quote counter climbing as it collects evidence, and the switch into the answering phase once it ends browsing. This is the actual "agent loop" WebGPT runs, four years before "ReAct" was a term anyone used.

Rejection sampling, visualized: n independent full episodes are sampled from the same policy, each scored by the reward model, and only the highest-scoring one is kept. Bigger n costs more inference compute but reliably raises quality — this is the mechanism behind the paper's biggest single quality gain (68% preferred over plain BC at n=64).

The algorithm, simplified

# The core browsing loop. One call to the policy per step; state is fully re-serialized
# as text each turn (no hidden memory) — this is what makes it a normal LM completion.

def browse_and_answer(question, policy, browser, reward_model=None, max_actions=100):
    references = []                       # quotes collected so far; these become citations
    history = []                          # human-readable log of past commands, re-shown each turn
    page = browser.search(question)       # start with an initial search, like a human would

    for step in range(max_actions):
        obs = render_state(question, page, history, references, max_actions - step)
        command = policy(obs)             # ONE completion: "Search x" / "Clicked on 3" / "Quote: ..." / "End: Answer"
        history.append(command)

        if command.startswith("End"):
            break
        page = browser.step(command, page)   # search / click / scroll / find, mutates `page`
        if command.startswith("Quote"):
            text = command.removeprefix("Quote: ")
            if text in page.visible_text:     # can only quote what's actually on screen
                references.append((page.title, page.domain, text))

    if not references:
        return None                        # no evidence collected -> no answer allowed

    answer_prompt = format_answer_prompt(question, references)
    answer = policy(answer_prompt)         # SEPARATE completion: compose the cited answer
    return answer, references


def best_of_n(question, policy, browser, reward_model, n=64):
    # Rejection sampling: this IS the paper's strongest optimization method, and it's this simple.
    candidates = [browse_and_answer(question, policy, browser) for _ in range(n)]
    candidates = [c for c in candidates if c is not None]
    scores = [reward_model(question, answer, refs) for answer, refs in candidates]
    best = candidates[scores.index(max(scores))]
    return best

Built on Prior Work

Results & Evidence

• ELI5 vs. human demonstrators: the 175B best-of-64 model’s answers preferred 56% of the time (vs. 50% = “no better than imitation”). This is the paper’s cleanest result because both sides used the same tool, the same detailed criteria, and the same style of answer.
• ELI5 vs. Reddit’s top-voted answer: preferred 69% of the time, using stripped citations and a separate, less-informed set of labelers for fairness. Big jump over Krishna et al.’s 23%, but the authors are explicit that this comparison is noisier — real Reddit answers aren’t trying to satisfy the same rubric, and un-cited WebGPT prose is stylistically distinguishable from Reddit prose even with citations removed (weaker blinding).
• TruthfulQA (adversarial short-form): WebGPT true 75% of the time, true-and-informative 54% — beats base GPT-3 at every size on both metrics, and (unlike GPT-3) truthfulness improves with model scale for WebGPT. Still below human performance.
• TriviaQA (short-form, zero real training focus): with a small extra fine-tune step to convert WebGPT’s long answers into short-form ones, 175B WebGPT+GPT-3 hits 69.5% overall accuracy, roughly matching UnitedQA-E (68.9%) despite WebGPT never being built for this task and being trained on only 143 TriviaQA demonstrations.
• Rejection sampling vs. RL, head to head: best-of-64 BC preferred 68% of the time over plain BC; RL alone preferred only 58% of the time over BC; combining RL + rejection sampling adds almost nothing over rejection sampling alone. The paper’s own explanation: both optimize the same reward model, so both are vulnerable to overoptimizing it, but RL is worse at this because it reduces policy entropy (less diverse exploration) and the reward model was trained mostly on BC/rejection-sampled data, not RL data.
• Scaling: doubling demonstrations raises the RM score of the policy by ~0.13; doubling comparisons raises RM accuracy by ~1.8%; doubling policy parameters raises RM score by ~0.09; doubling RM parameters raises RM accuracy by ~0.4%. Roughly: more comparison data (cheap-ish, 10 min/label) matters more than more demonstration data (expensive, 15 min/label) or more parameters, at the margins they tested.

What this does NOT establish:

• No public numeric comparison to GPT-3 (no browsing) on ELI5-style human preference — only on TruthfulQA. It’s plausible some of the ELI5 gains come from fine-tuning/format effects that have nothing to do with browsing; the paper doesn’t isolate that ablation.
• The reward model is trained on human labelers judging “is this supported by the reference,” not “is this actually true” — so the entire system is only as truthful as its labelers’ ability to evaluate source trustworthiness (the paper admits it made “a number of difficult judgment calls” here it doesn’t expect universal agreement with).
• The comparison to Reddit’s top answer uses different labelers with different instructions than the demonstration comparison — the 69% number and the 56% number are not measuring quite the same thing, despite being presented side by side.
• Small-sample bias experiments (60 hand-written questions for question-stance, 64 answers for reference-point bias) are explicitly flagged by the authors as too small to draw firm conclusions from, just suggestive.
• The action space explicitly forbids write-access to the web (no forms, no editing Wikipedia) — the safety story about “risk of live web access” is narrower than “an LLM agent with browser access” in general; a browsing agent with write actions is a materially different risk profile.

How You’d Use It

This is the direct blueprint for any “cited/grounded answer” product you’d ship — a research assistant for your users, an internal knowledge-base Q&A bot, or a support-ticket answering agent where “it made that up” is the failure mode that kills trust.

• Your applications — force evidence collection as a side effect of tool use, not as a separate “please cite your sources” instruction. A model told to answer-then-cite will confabulate citations; a model that can only claim what it quoted during a real tool call is structurally more honest. This is the single most portable idea in the paper and works today with a normal ReAct/tool-calling loop — you don’t need OpenAI’s exact browsing environment, you need the same design constraint.
• Your workflows — turn “is this true” into “is this supported” for any human-in-the-loop QA/review pipeline. If you’re building an eval harness or a labeling workflow for an agent’s outputs, judge claim-to-source support, not raw factual accuracy — it’s faster, cheaper, and gets much higher inter-labeler agreement (this paper measured 74% vs. what open-ended fact-checking would likely give).
• Your harness — rejection sampling is the cheap RLHF you can ship today. If you want “optimize against my reward signal” but don’t want to run PPO, best-of-n against any scorer (a reward model, an LLM judge, a rule-based verifier) gets most of the win in this paper’s own head-to-head, with zero extra training infrastructure — just more inference spend at generation time. This is a cheap lever worth reaching for before you build any training pipeline.
• Your harness — stateless-per-turn agent design (re-serialize a summary each step instead of relying on the model’s running context) is a useful pattern for long-horizon browsing/tool agents where context length or attention degradation over long transcripts is a real production problem.

Build Your Own (Minimal Recipe)

You can get a meaningful fraction of WebGPT’s value without OpenAI’s training budget. Minimal stack:

1. A tool-use loop, not a custom browser. Use any modern LLM with function/tool calling (search, fetch-and-simplify-page, quote) instead of building a bespoke text-browser DOM. Restrict the “quote” tool so it only succeeds if the exact text is present in the last fetched page — that constraint is what makes citations trustworthy, and it costs nothing to implement.
2. Skip BC — a strong instruction-tuned model already knows how to use tools. The BC step existed because 2021 GPT-3 had never seen a tool-calling format; today’s frontier models don’t need that step. Start directly from prompting + tool schemas.
3. Collect comparisons, not demonstrations, as your first data investment. The paper’s own scaling results say comparisons move the needle more per unit effort than demonstrations. A few hundred to low-thousands of “which cited answer is better” preference pairs (from your own team, or an LLM-judge as a bootstrap) gets you a usable reward model or, more simply, a good rubric for an LLM-as-judge.
4. Ship rejection sampling before RL. Sample n=4–16 full answer+citation episodes, score with an LLM judge or a lightweight reward model, keep the best. This is a day of engineering, not a training run, and the paper’s own numbers say it beats RL for this task.
5. The one genuinely hard part: building a judge (human or model) that reliably scores “claim supported by source X” rather than “sounds right.” This is where WebGPT’s real engineering effort went (labeler instructions, annotation UI, agreement-rate monitoring) — skimp here and rejection sampling just learns to pick the most confident-sounding hallucination instead of the best-supported one.
6. Libraries/models to reach for: any tool-calling-capable model (Claude, GPT) + a web search API (Bing/Brave/Serper/Tavily) + Readability.js-style HTML-to-text simplification (or trafilatura/readability-lxml in Python) + a small preference-labeling UI (even a spreadsheet works at low volume) + an LLM-judge prompt as reward-model stand-in for a first pass.

How to Improve It

• Debate/self-critique the citations, not just the answer. The paper itself proposes this: train the model to also argue against its own cited claims (in the spirit of AI-safety-via-debate / recursive reward modeling), which would directly attack the “cherry-picked convincing sources” failure mode they flag in §6.4.
• Ablate browsing itself. Run a matched GPT-3-no-browsing RLHF baseline (Stiennon-style summarization setup, but for ELI5) to isolate how much of the 56%/69% wins come from the browser versus from human-feedback fine-tuning in general. The paper never runs this control.
• Multi-hop verification pass. Add a second browsing episode whose only job is to re-verify each quote’s claim against 1–2 independent sources before the answer is finalized — attacks non-imitative falsehoods (paraphrase/synthesis errors) directly, which the paper admits it couldn’t even measure well with labelers.
• Adversarial-question training data, as the paper suggests for TruthfulQA-style distribution shift: mine or generate conspiracy/misconception-adjacent questions specifically, since the question-stance experiment (§H.1, small-n but suggestive) shows the model is more likely to be wrong when the question itself implies a false belief.
• Decompose the labeling task. The paper explicitly flags that ~15-minute demonstrations and ~10-minute comparisons are longer than “conventional wisdom” labeling tasks and speculates decomposition could help; a smaller, faster micro-task (e.g., “is claim X supported by quote Y: yes/no”) could 5-10x your labeled-data throughput per dollar for anyone rebuilding this reward-model pipeline.

Glossary

• LFQA (long-form question answering) — answering with a paragraph, not a single fact/span.
• Behavior cloning (BC) — supervised fine-tuning on human demonstrations; the model copies the human’s actions directly, no notion of “better.”
• Reward model (RM) — a model trained to output a single score for a (question, answer) pair such that score differences predict human preference.
• Bradley–Terry / Elo-style loss — the specific way “preference probability = sigmoid(score A − score B)” is turned into a training loss (cross-entropy on which side humans picked).
• PPO (Proximal Policy Optimization) — the RL algorithm used to fine-tune the policy against the reward model, with clipping to prevent destructively large policy updates in one step.
• KL penalty — a regularization term that penalizes the RL policy for drifting far (in distribution) from the original BC model, at every generated token; prevents “reward hacking” the reward model into gibberish.
• Rejection sampling / best-of-n — sample n full outputs, score them, keep only the highest-scoring one; an inference-time (not training-time) way to optimize against a scorer.
• Imitative falsehood — a false statement the training objective actually rewards (e.g., repeating a common misconception because it’s common in the data), even with infinite data/compute.
• Non-imitative falsehood — a false statement caused by the model failing at its objective (most hallucinations); it’s not what training “wanted,” it’s a mistake.
• Automation bias — the human tendency to over-trust a system’s output because it appears confident/authoritative (a specific risk the paper flags for citation-heavy answers).
• ReAct-style loop — an “observe current state → decide one action → act → get new observation” cycle; WebGPT implements this pattern via fine-tuning + RL rather than pure prompting (which is how the term is usually used today).
• ELI5 dataset — questions from Reddit’s “Explain Like I’m Five” subreddit, used as the main training/eval source; answers are graded against the community’s top-voted response.
• TruthfulQA — an adversarially constructed benchmark of short questions designed to elicit common human misconceptions, scored on truthfulness and informativeness.