Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

TL;DR

Traditional LLM benchmarks (MMLU, HELM) measure narrow, closed-ended knowledge and completely miss what makes a chatbot feel good to a user: following instructions across a multi-turn conversation. So how do you score that at scale without paying humans to rank thousands of conversations? This paper’s answer is LLM-as-a-judge: hand a powerful model two answers (or one) and ask it to pick the better one, with a written rationale. The authors stress-test this idea — they find judges have position bias (favor whichever answer is shown first), verbosity bias (favor longer answers), and self-enhancement bias (favor their own outputs) — then show simple fixes (swap positions, give a reference answer, use chain-of-thought). The headline result: GPT-4 as judge matches human expert preference 85% of the time (vs. 81% human-human agreement) on their new MT-Bench, and matches crowd preference on Chatbot Arena. This paper is why “LLM-as-a-judge” and the Arena Elo leaderboard are now standard infrastructure.

Problem & Motivation

The concrete pain: you cannot tell a good chatbot from a bad one with the benchmarks everyone was using. RLHF-aligned models (the ones users overwhelmingly prefer) score about the same on MMLU and HELM as the raw, un-aligned base models they were built from. The paper opens with a brutal example (Figure 1): base LLaMA-13B and fine-tuned Vicuna-13B both look competitive on a multiple-choice economics question, but on the open-ended follow-up (“how does this affect my daily life? give 3 examples”) LLaMA produces a useless repetitive loop while Vicuna gives three real examples. A multiple-choice benchmark scores these as roughly equal. A human (and GPT-4) instantly sees Vicuna won.

Why prior approaches fall short:

• Closed-ended benchmarks (MMLU, HellaSwag, GSM-8K) test short, auto-checkable answers. They don’t measure multi-turn instruction following or open-ended helpfulness — exactly the things users care about.
• Reference-overlap metrics (BLEU, ROUGE) need a gold answer to compare against. Open-ended questions (“write a travel blog about Hawaii”) have no single correct answer, so string-overlap is meaningless.
• Human evaluation is the gold standard but is “exceptionally slow and costly.” You can’t run it every time you tweak a prompt or fine-tune a checkpoint.

The gap, stated in one sentence: there was no scalable, automated way to measure whether a chatbot’s open-ended answers align with human preference.

What’s New (Core Contribution)

Four things, three of which became industry infrastructure:

1. A systematic study of LLM-as-a-judge. Before: people (including these authors’ own blog post) had used a big LLM to grade answers ad hoc. Now: a controlled measurement of how well it actually agrees with humans and which biases corrupt it, with quantified fixes. This turned a folk technique into a validated method.

2. MT-Bench — Before: no compact benchmark for multi-turn, open-ended chat. Now: 80 hand-written two-turn questions across 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, humanities) designed both to mimic real use and to separate strong from weak models.

3. Chatbot Arena — Before: no way to gather preference data “in the wild.” Now: a crowdsourced platform where users chat with two anonymous models side-by-side and vote. This became the Arena Elo leaderboard the whole field now cites.

4. Named and measured the three judge biases (position, verbosity, self-enhancement) plus a reasoning limitation, and gave cheap mitigations. This is the genuinely transferable engineering content — it’s what you actually implement.

The data release (80 MT-Bench questions, 3K expert votes, 30K human-preference conversations) is itself a contribution that fueled follow-on work.

How It Works (Technically)

There’s no neural-network novelty here — the “model” is just GPT-4 with a carefully designed prompt. The intellectual content is in the evaluation protocol and the bias mitigations. Let’s go through the mechanism.

The three judge modes

1. Pairwise comparison — Judge sees (question, answer_A, answer_B) and returns “A wins” / “B wins” / “tie”. Most reliable for ranking, but the number of pairs grows quadratically with the number of models (n models → n(n−1)/2 pairs), so it gets expensive at scale.
2. Single-answer grading — Judge sees (question, answer) and returns an absolute score (e.g., 1–10). Scales linearly, but absolute scores drift if you change the judge model, and it can miss subtle differences between two close answers.
3. Reference-guided grading — Same as above but you also paste in a reference solution. Crucial for math/reasoning where the judge needs to know the right answer.

The biases, demystified

This is the heart. Each bias is a measured failure mode of the judge:

Position bias — the judge favors whichever answer appears first. They tested it by generating two near-identical answers (call GPT-3.5 twice at temperature 0.7) and seeing whether the verdict flips when you swap the order. “Consistency” = % of cases where the verdict is the same in both orders. The numbers:

Read that top row: Claude was consistent only 24% of the time and slammed the first answer 75% of the time. Even GPT-4, the best, flips its verdict on a third of swaps. The authors suspect this is baked into the left-to-right causal-transformer architecture or the training data, but leave the root cause as future work.

Verbosity bias — the judge favors longer answers even when length adds nothing. They built a “repetitive list attack”: take an answer with a 5-item list, ask GPT-4 to rephrase those 5 items (no new info) and prepend them, making a 10-item list that says the same thing. “Failure” = judge prefers the bloated version. Result: Claude-v1 and GPT-3.5 fell for it 91.3% of the time; GPT-4 only 8.7%. So verbosity bias is real but GPT-4 mostly resists it.

Self-enhancement bias — judges may favor their own outputs. GPT-4 gave itself a ~10% higher win rate than humans did; Claude gave itself ~25% higher. But they’re honest: the sample is small and they “cannot determine” it’s a true effect, because you can’t cleanly rephrase an answer into another model’s style without changing quality. This is the most carefully hedged claim in the paper.

Limited math/reasoning grading — even when GPT-4 can solve a math problem on its own, showing it a wrong answer in the judge prompt often misleads it into endorsing the wrong answer. The context poisons the judge.

The fixes

• Swap positions (the fix for position bias): call the judge twice with the two answers in both orders. Only declare a winner if it wins both times; otherwise call it a tie. This is the “conservative” approach they use throughout. (The “aggressive” alternative is to randomize position and trust it averages out at scale.)
• Few-shot judge: add 3 example judgments to the prompt. Pushed GPT-4 consistency from 65% → 77.5% — but made API calls 4× more expensive and risks introducing new biases, so they don’t use it by default.
• Chain-of-thought judge: ask the judge to solve the problem itself before grading. Helps, but the judge often makes the same mistake as the answer it’s grading.
• Reference-guided judge (the math fix): first ask the judge to solve the problem independently, then inject that solution as a reference into the grading prompt. This dropped the math grading failure rate from 70% → 15%.

Multi-turn judging

MT-Bench questions have two turns, so a pairwise comparison means juggling 2 questions and 4 answers. They found that splitting the conversation into separate prompts makes the judge lose track of which earlier answer belongs to whom. Fix: paste the complete two-turn conversation into one prompt and tell the judge to focus on the second turn.

Architecture & data flow

flowchart TD
  Q[MT-Bench question or Arena chat] --> MA[Model A answer]
  Q --> MB[Model B answer]
  MA --> J1[Judge call: A first, B second]
  MB --> J1
  MA --> J2[Judge call: B first, A second]
  MB --> J2
  J1 --> AGG{Agree in both orders?}
  J2 --> AGG
  AGG -->|yes| WIN[Declare winner]
  AGG -->|no, flipped| TIE[Call it a tie - position bias]
  REF[Optional: judge-generated reference answer] -.-> J1
  REF -.-> J2
  WIN --> ELO[Aggregate into win-rate / Elo leaderboard]
  TIE --> ELO

Interactive: toggle which answer is shown first and watch a biased judge flip its verdict. The "swap-and-require-agreement" fix turns flipped verdicts into honest ties. Drag the bias slider to see how a stronger judge (lower bias) flips less often.

Interactive: agreement between GPT-4-judge and humans rises as the two models being compared get further apart in quality (bigger win-rate gap → easier call → higher agreement, ~70% up to ~100%). Schematic, built to match the paper's reported trend.

The algorithm, simplified

# Pairwise LLM-as-a-judge with the position-bias fix.
# llm(prompt) -> str ; we parse "A", "B", or "tie" from the verdict.

JUDGE_PROMPT = """You are an impartial judge. Given a question and two
assistant answers, decide which answer better follows the user's
instructions and is more helpful. Output exactly one of: [[A]], [[B]], [[C]]
(C = tie). Then explain briefly."""

def judge_once(question, ans_first, ans_second, reference=None):
    ref = f"\nReference answer:\n{reference}" if reference else ""
    verdict = llm(f"{JUDGE_PROMPT}{ref}\n\nQuestion: {question}\n"
                  f"Answer A: {ans_first}\nAnswer B: {ans_second}")
    return parse(verdict)            # -> "A" | "B" | "tie"

def judge_pairwise(question, ans_X, ans_Y, reference=None):
    # call twice, swapping order, to neutralize POSITION BIAS
    v1 = judge_once(question, ans_X, ans_Y, reference)   # X first
    v2 = judge_once(question, ans_Y, ans_X, reference)   # Y first
    # map each verdict back to the actual model (X / Y), not the slot (A / B)
    win1 = {"A": "X", "B": "Y", "tie": "tie"}[v1]
    win2 = {"A": "Y", "B": "X", "tie": "tie"}[v2]
    if win1 == win2 and win1 != "tie":
        return win1                  # consistent winner across both orders
    return "tie"                     # disagreement => the bias showed up => tie

# For math/reasoning, generate the reference FIRST, then judge with it:
def judge_math(question, ans_X, ans_Y):
    reference = llm(f"Solve this problem step by step:\n{question}")
    return judge_pairwise(question, ans_X, ans_Y, reference=reference)

That’s essentially the whole method. The [[A]] double-bracket convention is a real parsing trick from the paper — it makes the verdict easy to extract with a regex.

Built on Prior Work

The deep insight that makes the whole thing work: an RLHF-aligned LLM has already internalized human preference, so asking it to judge is reusing that alignment rather than building a new reward model from scratch.

Results & Evidence

Headline (MT-Bench, expert humans): Under setup S2 (non-tie votes only), GPT-4-judge agrees with human experts 85% of the time — higher than human-human agreement at 81%. GPT-4 single-answer grading agrees with GPT-4 pairwise at 85% too, meaning GPT-4 has a stable internal rubric whether grading absolutely or comparatively.

Persuasion effect: When a human disagreed with GPT-4, they were shown GPT-4’s rationale; humans judged it “reasonable” in 75% of cases and changed their own vote 34% of the time. (Note: this cuts both ways — it could mean GPT-4 is right, or that humans are anchoring on a confident machine.)

Agreement scales with quality gap: Agreement between GPT-4 and humans climbs from ~70% (close models) to ~100% (very different models). Easy calls are easy; the judge struggles exactly where humans also struggle.

Chatbot Arena (crowd): Same trend holds with real-world crowd votes. GPT-4 produces more affirmative (non-tied) verdicts than other judges, suffering less from position bias.

What the evidence does NOT establish — be honest:

• It’s GPT-4-specific. Claude and GPT-3.5 were much weaker judges (91% verbosity-attack failure, terrible position consistency). “LLM-as-a-judge works” really means “GPT-4 works”; weaker/open models in 2023 did not.
• Self-enhancement bias is unproven, by the authors’ own admission. If you use a model to judge a leaderboard that includes itself, you have an unresolved conflict of interest.
• Small, expert-skewed human panel. 58 labelers, mostly grad students — more expert than a typical crowd, so 81% human-human agreement may be optimistic for general users.
• The judge inherits the judge’s blind spots. It can’t reliably grade math/reasoning it can’t do, and it can be misled even on problems it can do.
• “Agreement with humans” ≠ “correct.” If humans are biased (e.g., toward verbosity), a judge that agrees with them is faithfully reproducing that bias, not eliminating it.

How You’d Use It

For an AI services company, this paper is a toolbox, not just a result. Concrete slots:

• Regression eval for client chatbots. Every time you change a system prompt, swap a model, or fine-tune, run a held-out set of representative client questions through pairwise LLM-as-a-judge (new version vs. current production). You get a win-rate delta in minutes instead of a human review cycle. This is the single highest-ROI use.
• Offer it as a deliverable. “Automated quality evaluation harness” is a sellable artifact: a curated question set in the client’s domain + an LLM-judge pipeline + a dashboard of win rates over time. Clients can’t easily build the methodology themselves.
• Multi-agent orchestration (your ARC MAS background). Drop a judge agent into the loop as a critic/router: have it score candidate outputs from worker agents and pick the best, or gate a response before it reaches the user. The position-swap and reference-guided tricks port directly into the critic’s prompt.
• A/B and model-selection decisions. Use single-answer grading to cheaply triage many models/prompts, then pairwise-judge the finalists. This matches the paper’s own pairwise-for-precision, single-for-scale tradeoff.
• Synthetic preference data. The judge’s verdicts can become a preference dataset for fine-tuning a smaller, cheaper model — distilling GPT-4’s taste into something you can run at volume.

The realistic effort to stand up a basic version: a day or two. The hard part isn’t the code; it’s curating questions that actually represent your client’s use case.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

1. Question set (the real work). 30–80 questions that mirror your client’s actual traffic. Borrow MT-Bench’s structure: spread across the capability categories that matter to them, include a few deliberately hard ones to separate models.
2. Answer generation. Run each question through the two candidate systems (e.g., prod vs. new prompt). Log full multi-turn conversations, not just final answers.
3. The judge call. Use the judge_pairwise function above with a strong judge (GPT-4-class or Claude-3.5+). Use the [[A]]/[[B]]/[[C]] verdict format for easy parsing.
4. The position-bias fix is non-negotiable. Always call twice and swap. Skipping this is the #1 way people get garbage eval numbers. This is genuinely the one hard/easy-to-miss part.
5. Reference-guided for anything with a correct answer (math, code, factual extraction): generate the reference first, then judge.
6. Aggregate. Win rate is enough to start. If you’re comparing many models, compute Elo from pairwise outcomes (the Arena approach) — pip install choix or a few lines of logistic regression over win/loss pairs.

Libraries/models to reach for: the official lm-sys/FastChat llm_judge module ships the exact prompts and harness; any OpenAI/Anthropic SDK for the calls; choix or trueskill for ratings. You do not need to train anything.

The two genuinely hard parts: (1) writing questions that discriminate (too easy → everything ties), and (2) trusting the numbers only on large quality gaps — near-tie verdicts from the judge are noisy.

How to Improve It

Limitations as leverage — five testable directions:

1. Eliminate self-judging conflict of interest. Use a panel of diverse judges (GPT-4 + Claude + Gemini), majority-vote, and explicitly exclude any judge from grading its own family. Test whether panel agreement with humans beats any single judge. (This became “LLM-jury” in later literature — you can build and validate it.)
2. Calibrate out verbosity bias directly. Add length as a control: either normalize scores by answer length, or instruct the judge to ignore length and penalize redundancy, then re-run the repetitive-list attack to measure residual bias.
3. Tool-augment the judge for math/code. Instead of (or in addition to) a reference answer, give the judge a code interpreter / calculator so it executes rather than guesses. Should crush the math grading failure rate below 15%.
4. Fine-tune a small open judge. The paper’s Appendix F teases a fine-tuned Vicuna-13B judge. Distill GPT-4 verdicts into a 7–13B model so you can judge at production volume and cost. Measure the agreement drop vs. the cost saving — likely a great tradeoff for high-volume clients.
5. Detect, don’t just average, low-confidence calls. Have the judge emit a confidence and route only the close/uncertain pairs to humans (active learning). This gets you near-human accuracy at a fraction of the human cost, and is directly sellable as a “human-in-the-loop only when it matters” pipeline.

Glossary

• LLM-as-a-judge — using a strong LLM to grade or compare other models’ answers, as a stand-in for human preference judgment.
• MT-Bench — the paper’s 80-question, two-turn benchmark for multi-turn instruction-following chat.
• Chatbot Arena — the crowdsourced platform where users vote between two anonymous chatbots; source of the Arena Elo leaderboard.
• Position bias — a judge’s tendency to favor whichever answer is presented first (or in a fixed slot).
• Verbosity bias — a judge’s tendency to prefer longer answers even when length adds no real information.
• Self-enhancement bias — a judge’s possible tendency to rate its own outputs higher (unproven here, by the authors’ admission).
• Pairwise comparison — judge picks the better of two answers; scales quadratically in number of models.
• Single-answer grading — judge assigns an absolute score to one answer; scales linearly but drifts.
• Reference-guided grading — judging with a known-good answer pasted in; the fix for math/reasoning.
• Chain-of-thought (CoT) — prompting the model to reason step-by-step before answering/grading.
• RLHF — Reinforcement Learning from Human Feedback; training that aligns a model to human preferences, which is why an aligned model makes a usable judge.
• Agreement — probability two randomly chosen judges (of given types) pick the same winner on a random question; the paper’s core metric.
• Elo — a rating system (from chess) that converts pairwise win/loss outcomes into a single skill number per model.
• MMLU / HELM — standard closed-ended capability benchmarks the paper shows can’t distinguish aligned chatbots from base models.