Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvement

TL;DR

Evaluating LLM outputs is the real bottleneck now that generation is cheap. The usual options are all bad: humans are slow and expensive, word-overlap metrics (ROUGE, BLEU) miss meaning, and “LLM-as-judge” spits out a single opaque number you can’t debug. This paper’s fix, BinEval, decomposes each evaluation criterion into a set of atomic yes/no questions, has an LLM answer each one independently, and aggregates the answers into a score per dimension (coherence, consistency, etc.). Because every score is just “fraction of questions answered yes,” you can see which checks failed and why. Across three standard benchmarks (SummEval, Topical-Chat, QAGS) it matches or beats strong baselines like G-Eval and UniEval — with a big edge on factual consistency — and the same question-level feedback drives an iterative loop that improves both evaluator prompts and generator prompts. It needs no training and no task-specific setup.

Problem & Motivation

Here is the concrete pain. You are iterating on a summarization prompt for a client. You run an LLM judge and it says “3 out of 5.” Now what? Is the summary factually wrong? Missing key content? Awkwardly written? Off-topic? The single number tells you nothing you can act on. You are flying blind on the exact thing you are paid to improve.

Every prior approach falls short in a specific way:

• Human evaluation is the gold standard but is slow, expensive, and doesn’t scale to the dozens of iterations a real build needs.
• Lexical metrics (ROUGE, BLEU, METEOR) count overlapping words against a reference. They reward surface similarity and completely miss whether the output is true or means the same thing. A factually wrong summary that reuses the article’s words scores well.
• Embedding metrics (BERTScore, MoverScore) compare meaning in vector space but still produce one fuzzy similarity number with no diagnosis.
• Holistic LLM judges (G-Eval, Prometheus) are the current best. But they return an opaque scalar, they inherit position/verbosity/self-enhancement biases, and — the paper’s key empirical finding — they suffer ceiling effects: they cram most outputs into the top of the scale and can’t tell a mediocre summary from a clearly flawed one. Worse, they conflate “sounds plausible” with “is correct.” The paper shows a summary with three real factual errors that G-Eval and UniEval both rate a perfect 5.0 for consistency, because it reads fluently.

The premise is almost embarrassingly simple: a hard judgment (“rate factual consistency 1–5”) is easier and more reliable if you break it into small checkable questions (“Are all named entities accurate?” “Is anything fabricated?”). Small questions are answerable; big judgments are guesses.

What’s New (Core Contribution)

Four contributions, each a “before → now”:

• Decompose the criteria, not the content. Before: FActScore/RAGAS-style methods break the generated text into atomic facts to verify. Now: BinEval breaks the evaluation rubric itself into atomic yes/no questions, generated by a single task-agnostic meta-prompt. The same meta-prompt handles summarization, dialogue, or instruction-following — only the task description changes.
• Interpretable, multi-dimensional scores for free. Before: a judge emits one number (G-Eval) or fine-tunes a model per task (UniEval trains a T5). Now: every score is literally “yeses ÷ questions,” grounded in individual verdicts each with a natural-language explanation. No training, fully inspectable.
• Question-level feedback as an optimization signal. Before: prompt optimizers (DSPy, OPRO, APE) tune prompts against a scalar reward. Now: BinEval’s per-question pass/fail identifies exactly which criterion failed, and a two-phase loop uses that to rewrite either the evaluator’s prompt (to match humans/a stronger model) or the generator’s prompt (to fix its outputs).
• Cross-model alignment via disagreement. Genuinely novel: because both a strong and a weak evaluator answer the same binary questions, you can diff them question-by-question. The disagreements pinpoint which criteria a weaker/cheaper model judges inconsistently, giving a targeted way to align it to a stronger reference (useful when migrating between model families).

An honest read: the “decompose then verify” idea is not new (FActScore, and UniEval already reformulated evaluation as Boolean QA). What is new is (a) decomposing the rubric generically with one meta-prompt across dimensions, (b) using multiple binary questions per dimension rather than UniEval’s single Boolean, and (c) turning the question-level disagreement signal into a prompt-optimization loop. The gains over UniEval come mostly from “several targeted questions” beating “one coarse yes/no.”

How It Works (Technically)

BinEval has three parts: (1) generate the questions, (2) answer and aggregate them into scores, (3) use the answers to iteratively improve prompts. Let me demystify each, then trace one real example end to end.

Part 1 — Binary Question Generation

A task prompt T (e.g., a summarization instruction) is turned into a set of binary questions by an LLM guided by a meta-prompt M:

Q = F_LLM(T; M) = {q1, q2, ..., qN}

Plain English: “feed the task description into an LLM and ask it — using a fixed instruction template — to write a checklist of yes/no questions.” The meta-prompt runs a two-step decomposition:

• Step 1 – Summarize. First rewrite T into an explicit list of requirements R = {r1, ..., rK} (e.g., “must include the key event,” “must not add unsupported claims,” “must be one coherent narrative”). This forces the model to form a complete picture of the task before splitting hairs.
• Step 2 – Decompose. For each requirement rk, write one or more binary questions where yes = requirement satisfied, no = violated. Requirements that secretly bundle several checks get split into separate questions, and each question is paired with a short example of what a “no” looks like (to sharpen the negative case).

The questions are grouped into evaluation dimensions D (coherence, consistency, fluency, relevance):

Q = ⋃(d∈D) Qd where Qd is the questions for dimension d.

That is just: “the full checklist is the union of the per-dimension sub-checklists.” The meta-prompt is task-agnostic — the same M produces sensible questions for any task; only T changes.

Part 2 — Binary Evaluation and Scoring

Given an evaluator LLM E, an input x (source document / transcript / instruction), an output y (the summary / reply / completion), and a question qi, define:

fE(x, y, qi) ∈ {0, 1} — 1 if the evaluator answers “yes,” 0 if “no.”

Alongside each 0/1 the evaluator also writes a natural-language explanation ei — this is what makes it debuggable. The per-dimension score is simply the fraction of that dimension’s questions answered yes:

Sd(x, y) = (1 / |Qd|) · Σ(qi∈Qd) fE(x, y, qi)

And the overall score is the fraction across all N questions:

S(x, y) = (1 / N) · Σ(i=1..N) fE(x, y, qi)

Both live in [0, 1], where 1 = every criterion satisfied. To compare against benchmarks that use a 1–5 Likert scale, rescale with a straight-line (affine) map:

S'(x, y) = S(x, y) · (b − a) + a

That’s it — no learned aggregation, no weights. “Count the yeses, divide, stretch to the target scale.” The simplicity is the point: the score is transparent by construction.

A concrete trace (the consistency example from the paper)

Task: judge whether a summary is factually consistent with its source article. The summary looks clean — right aircraft (RC-135U, SU-27), right event — but has three planted errors: it misattributes Russia’s stated purpose to the Pentagon, invents a dailycaller.com URL not in the source, and conflates the two sides’ accounts.

BinEval (with Claude as the evaluator) decomposes consistency into 7 binary questions and answers each:

Score = 3 yes / 7 = 0.43, rescaled to 1–5 → 1.57. The human rating was 2.0 (off by only 0.43). Meanwhile G-Eval and UniEval both said 5.0 — perfectly consistent — because the text is surface-plausible. Holistic scoring rewarded fluent, topically-correct text even though specific claims were false. The decomposition caught 4 of the 7 error types as separate probes.

The claim-level probe in action. Toggle each of the 7 consistency questions between yes/no and watch the aggregated BinEval score move; the dashed line is the human rating, and the flat bars show how the holistic judges (G-Eval, UniEval) pin this output at a perfect 5.0. Numbers are from the paper's Figure 4 example.

Part 3 — Iterative Prompt Optimization

The same yes/no signal drives a self-improvement loop, in two flavors.

Cross-Model Prompt Update (align a weak evaluator to a strong one). Let Esrc be a strong reference evaluator (Claude Sonnet 4) and Etgt a weaker target (gpt-oss-120b) whose prompt PE you want to improve. Each round:

1. Evaluate. Both models answer all questions on each test case: Asrc_j and Atgt_j.
2. Identify disagreements. Δj = {qi : Asrc_j(qi) ≠ Atgt_j(qi)} — literally the questions where the two models gave different yes/no answers. This is the magic: unlike diffing two scalar scores, this tells you which specific criterion they judge differently.
3. Extract lessons. A note-taker LLM reads each disagreement in context and writes a generalized lesson (e.g., “omission ≠ inconsistency; only penalize claims that are present but unsupported”).
4. Deduplicate. Merge lessons that say the same thing, keep unique ones: Dedup(ℓnew, M) = merge if similar to an existing lesson, else add.
5. Update prompt. An updater LLM finds the relevant substring sk in the current prompt and rewrites it to s'k: PE ← PE.replace(sk, s'k). Surgical edits, not a full rewrite.

Loop until the target’s per-dimension scores match the source within tolerance ε on all dimensions: |Sd_tgt − Sd_src| < ε.

Self Prompt Update (improve a generator against the evaluator). Same machinery, but now you improve a generation prompt PG:

1. Generate. y_j = LG(x_j; PG) — produce outputs with the current prompt.
2. Evaluate. Score them and collect the failing questions: Ej = {(qi, ei) : fE(x_j, y_j, qi) = 0} — every check the output flunked, plus the explanation.
3. Extract lessons from those failures with the note-taker.
4. Dedup and rewrite PG the same way.

Loop until no evaluation errors remain or you hit a max-iteration cap.

Architecture & data flow

flowchart LR
  T[Task prompt T] --> M[Meta-prompt M]
  M -->|Step 1: summarize| R[Requirements r1..rK]
  R -->|Step 2: decompose| Q[Binary questions q1..qN<br/>grouped by dimension]
  X[Input x + Output y] --> E[Evaluator LLM E]
  Q --> E
  E -->|per question: 0/1 + explanation| V[Verdicts]
  V --> SD[Per-dimension score Sd<br/>= yeses / questions]
  SD --> S[Overall score S<br/>rescaled to 1-5]

flowchart TD
  START[Current prompt] --> EVAL[Answer all binary questions]
  EVAL --> SIG{Signal source}
  SIG -->|self-update| FAIL[Collect failing questions<br/>+ explanations]
  SIG -->|cross-model| DIS[Diff vs strong model:<br/>disagreeing questions]
  FAIL --> NOTE[Note-taker LLM<br/>extract lessons]
  DIS --> NOTE
  NOTE --> DEDUP[Deduplicate lessons]
  DEDUP --> UPD[Updater LLM<br/>rewrite prompt substrings]
  UPD --> STOP{Scores match / no errors<br/>/ max iters?}
  STOP -->|no| EVAL
  STOP -->|yes| DONE[Improved prompt]

Why decomposition actually works (three mechanisms)

The paper does the honest thing and asks why many-small beats one-big, identifying three effects and checking each on SummEval:

1. Complexity reduction. Each binary question isolates one verifiable property. “Are all named entities accurate?” is answerable; “rate consistency 1–5” is a guess. This alone can dominate — on consistency, decomposition had the weakest variance/coverage benefit yet the largest gain over UniEval (+0.195 Spearman), meaning simply turning fuzzy factual judgment into targeted sub-checks did most of the work.
2. Variance reduction via aggregation. Averaging N weakly-correlated yes/no classifiers cuts variance roughly like 1/N — the classic wisdom-of-many-weak-checks effect. It only pays off when the questions are not redundant. The paper measures inter-question correlation (phi coefficient): relevance/coherence questions are nicely decorrelated (phi ≈ 0.20, 0.28), fluency moderate (0.39), but consistency questions are highly correlated (phi ≈ 0.58) because “free of factual errors” and “no misrepresentation” overlap. So the averaging benefit varies by dimension — you can’t assume it.
3. Coverage of failure modes. Forcing an explicit checklist improves recall — you catch disjoint failures a single judgment blurs together. In fluency, “spelling” (Q2) and “punctuation” (Q3) are nearly uncorrelated (phi = 0.02) with different fail rates, so each catches errors the other misses.

Inter-question correlation (phi) within a dimension, from the paper's Figure 3. Blue = the questions overlap (redundant); pale = they probe genuinely different things. Switch between Consistency (highly correlated — averaging buys little) and Relevance (mostly independent — averaging genuinely reduces noise). This is how you'd audit whether your own generated checklist is pulling its weight.

The algorithm, simplified

# The whole of BinEval scoring: generate a checklist, answer it, average.
# Stubs: llm(prompt) -> str ; the interesting part is the decomposition + counting.

def build_questions(task_prompt, meta_prompt):
    # Step 1: turn the task into explicit requirements (one coherent picture first)
    requirements = llm(meta_prompt.summarize_step(task_prompt))   # -> ["include key event", ...]
    # Step 2: each requirement -> one or more yes/no questions, tagged with a dimension
    questions = []
    for r in requirements:
        for q_text in llm(meta_prompt.decompose_step(r)):         # split bundled reqs
            questions.append({"text": q_text, "dim": r.dimension}) # yes = satisfied
    return questions                                              # Q = {q1..qN} by dimension

def score(evaluator, x, y, questions):
    verdicts = []
    for q in questions:
        ans = evaluator(x, y, q["text"])        # -> {"yes": 0/1, "why": "..."} , answered INDEPENDENTLY
        verdicts.append((q["dim"], ans["yes"], ans["why"]))
    # per-dimension score = fraction of yeses in that dimension
    dims = {d: [v for (dim, v, _) in verdicts if dim == d] for d in set(q["dim"] for q in questions)}
    per_dim = {d: sum(v) / len(v) for d, v in dims.items()}
    overall = sum(v for (_, v, _) in verdicts) / len(verdicts)     # S in [0,1]
    return overall, per_dim, verdicts                             # verdicts = your debug trace

# Self-improvement loop: the failing questions ARE the feedback.
def optimize_generation(gen_prompt, cases, evaluator, questions, max_iter=3):
    for _ in range(max_iter):
        outputs = [llm(gen_prompt + x) for x in cases]                     # 1. generate
        fails = [(x, y, q["text"], a["why"])                               # 2. collect fails
                 for x, y in zip(cases, outputs)
                 for q in questions
                 for a in [evaluator(x, y, q["text"])] if a["yes"] == 0]
        if not fails: break
        lessons = dedup([note_taker(f) for f in fails])                    # 3. lessons
        gen_prompt = updater(gen_prompt, lessons)                          # 4. surgical rewrite
    return gen_prompt

Built on Prior Work

Results & Evidence

Setup. Two backbones — gpt-oss-120b (open) and Claude Sonnet 4 (strong) — temperature 0, averaged over two runs. Reported with Spearman ρ, Kendall τ, Pearson r correlation to human ratings at the summary level.

Evaluation quality (Part I).

• SummEval (1,600 human-annotated summaries, 4 dimensions): BinEval (Claude) is best overall — average ρ/τ of 0.563 / 0.491, beating G-Eval (GPT-4) at 0.514 / 0.418 and UniEval (T5) at 0.474 / 0.377. Biggest win is consistency (0.655). Under the same open backbone, BinEval (gpt-oss) beats both G-Eval and UniEval on average — the decomposition, not the model, is doing the work. UniEval (gpt-oss) collapses to 0.000 fluency correlation, showing a single yes/no is too coarse for a general model.
• Topical-Chat (dialogue): BinEval (Claude) best average ρ 0.632, strong on naturalness/engagingness. Decomposition transfers beyond summarization.
• QAGS (hallucination detection): BinEval (Claude) best average ρ 0.620; even the gpt-oss version substantially beats G-Eval (gpt-oss). This is the clearest case — factual consistency decomposes cleanly into claim checks.
• Distributional match: BinEval better matches human score distributions and avoids the ceiling effects (compressed top-of-scale) that plague UniEval/G-Eval, so it discriminates mid-tier from clearly-flawed outputs.

Iterative prompt update (Part II).

• SummEval evaluator prompts: self-update +0.075 average ρ, cross-model +0.070. Self-update helps most on fluency (+0.119); cross-model most on consistency (+0.136, the largest single gain). The two signals surface different error classes.
• IFBench generation prompts: self-update peaked at 38.0% (+3.4pp) at iteration 3 — then collapsed at iteration 4. Cross-model showed no gain.

Caveats — read these before you sell it.

• Relevance resists decomposition. Both update modes fail to improve it; over-decomposing relevance into “every actor, every motivation” makes the evaluator harsher than humans and destroys correlation (ρ dropped .505 → .357 in one run). Some human judgments are irreducibly holistic.
• Prompt bloat is real and dangerous. The IFBench prompt grew from 22 chars to 6,248 over 4 iterations; the accumulation of unusable instructions eventually degraded even categories that were working. Most gains land in the first 1–2 iterations; later iterations tend to hurt.
• Prompting can’t fix capability gaps. IFBench format/sentence constraints improved +17pp each (they’re “promptable”), but count/ratio/word constraints barely moved — “maintain a running counter” doesn’t give a model the ability to count. BinEval correctly diagnoses these but the fixes are unactionable.
• Cost. BinEval trades efficiency for diagnosis: it generates questions and answers each one, so many more model calls and more text processed than a single holistic judgment. On small workshop-scale benchmarks; no large-scale or multilingual validation here.

How You’d Use It

This is a genuinely useful building block for an AI services shop, and it slots in cleanly.

• Replace your opaque eval harness. If you run LLM-as-judge to gate client deliverables, swap the “rate 1–5” call for a BinEval checklist per quality dimension. You immediately get reasons to put in front of a client (“the summary failed ‘no fabricated content’ because of this URL”) instead of a naked number. That’s the difference between “trust me, it’s a 4” and an auditable QA report — a sellable artifact.
• Prompt tuning as a service. The self-update loop is a concrete, demonstrable way to improve a client’s generation prompt using its own failure checklist. You can show before/after with the exact lessons extracted. It is transparent enough to hand to a non-technical stakeholder.
• Cheap-model alignment. The cross-model trick is the commercially interesting one: use an expensive model (Claude) once as a reference to tune a cheap open model’s evaluator prompt until it agrees, then run the cheap model at volume. Same idea when a client migrates model families and wants their eval to stay stable.
• Agentic / multi-turn evaluation. In a multi-agent system, “where did the chain go wrong?” is the hard question. Claim-level binary checks on each agent’s output give you a step-level failure signal — far better for routing/repair than a trajectory-level score. The authors flag this as the natural extension, and it maps directly onto MAS debugging.

Realistic effort: a solid v1 is a few days. The value is high because interpretability is exactly what enterprise buyers ask for and most eval tools don’t give.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

1. Meta-prompt (the hard/creative part). Write one prompt that takes a task description and outputs a JSON list of yes/no questions tagged by dimension, each with a one-line “what a ‘no’ looks like.” Two steps inside it: first list requirements, then split each into questions. This prompt is your whole moat — spend your time here.
2. Evaluator call. For each question, call the LLM with (source, output, question) and force structured output: {"answer": "yes"|"no", "explanation": str}. Answer questions independently (separate calls or a strict per-question format) so one bad question can’t drag the others.
3. Aggregator (trivial). score = yeses / total, per dimension and overall; affine-rescale if you need 1–5. Keep the full verdict list — that’s your debug output.
4. Optimization loop (optional v2). Collect failing questions → note-taker LLM writes lessons → dedup by embedding similarity → updater LLM edits the prompt substring-by-substring. Cap it at 2 iterations and early-stop on a held-out set — this is where it breaks otherwise.

Reach for: any structured-output-capable model (Claude, gpt-oss); instructor/Pydantic or plain JSON mode for the 0/1 + explanation schema; a tiny embedding model for lesson dedup. No training, no GPUs.

The two genuinely hard parts: (a) writing a meta-prompt that generates non-redundant, well-covering questions — measure this with the phi-correlation check the paper gives you; and (b) knowing when to stop iterating before prompt bloat wrecks it.

How to Improve It

Limitations are the roadmap. Five concrete, testable ideas:

1. Weight the questions instead of plain averaging. The paper assumes fraction-satisfied maps linearly to quality and weights every question equally. Learn (or LLM-assign) per-question weights against a small human set — likely the fastest correlation win, especially on relevance.
2. Auto-prune redundant questions. Compute the phi matrix on a sample; drop or merge questions with high mutual correlation (consistency’s overlapping checks). Fewer, decorrelated questions = lower cost and better variance reduction. Directly attacks the cost caveat.
3. Route by dimension. Decompose the “promptable/decomposable” dimensions (consistency, fluency) but keep a holistic judge for the irreducibly-holistic ones (relevance). The paper’s own failure case proves forcing decomposition everywhere backfires — a hybrid should strictly dominate.
4. Add a bloat guard to the loop. Track prompt length and per-iteration held-out score; roll back any lesson that grows the prompt without improving the score. Turns the fragile 4-iteration collapse into a monotone improvement. Trivial to implement, high payoff.
5. Separate “promptable” from “computational” failures automatically. BinEval diagnoses count/ratio failures but can’t fix them by prompting. Detect these (e.g., the lesson mentions counting/arithmetic) and route them to a tool call or code-checker instead of stuffing the prompt — closes the gap on verifiable constraints.

Glossary

• BinEval — the paper’s method: decompose an eval criterion into yes/no questions, answer each, average.
• Holistic LLM judge — asking an LLM for one overall score (e.g., “rate 1–5”); the thing this paper replaces.
• Atomic binary question — a single yes/no check about one property, easy to answer reliably.
• Dimension — a quality axis being evaluated (coherence, consistency, fluency, relevance).
• Meta-prompt — the fixed instruction that turns a task description into the checklist of questions.
• Affine scaling — a straight-line rescale S·(b−a)+a to move a [0,1] score onto a 1–5 (or any) scale.
• Spearman ρ / Kendall τ / Pearson r — correlation measures; here, how well a method’s scores track human ratings (higher = better; ρ/τ care about ranking, r about linear fit).
• Ceiling effect — a scorer squashing most outputs into the top of the scale, so it can’t separate good from mediocre.
• Phi coefficient (φ) — correlation between two binary variables; here, whether two questions tend to fire together (redundant) or independently.
• Variance reduction via aggregation — averaging many weak, uncorrelated checks cancels their individual noise, roughly like 1/N.
• Cross-model update — using a strong evaluator’s answers as the reference to tune a weaker evaluator’s prompt via their per-question disagreements.
• Self-update — a model improving its own generation (or evaluation) prompt using its own failed checks as feedback.
• Note-taker / updater LLM — helper roles in the loop: one turns failures/disagreements into general lessons, the other edits the prompt to incorporate them.
• SummEval / Topical-Chat / QAGS / IFBench — benchmarks used: summarization quality, dialogue quality, hallucination detection, and instruction-following, respectively.
• UniEval / G-Eval — the main baselines: a fine-tuned Boolean-QA evaluator, and a chain-of-thought Likert LLM judge.