Self-Refine: Iterative Refinement with Self-Feedback

TL;DR

Large language models rarely nail an output on the first pass, but they’re surprisingly good at spotting what’s wrong with their own output if you just ask them to. Self-Refine turns that observation into an algorithm: generate an answer, prompt the same model to critique it, prompt the same model to rewrite it using that critique, and repeat. No reward model, no fine-tuning, no reinforcement learning — just three few-shot prompts and a loop. Across 7 tasks (dialogue, code optimization, code readability, math reasoning, sentiment reversal, acronym generation, constrained generation) with GPT-3.5, ChatGPT, and GPT-4, Self-Refine beat plain one-shot generation by roughly 5-40 percentage points, and the gains got bigger with stronger base models — meaning GPT-4 has more untapped headroom than its single-pass score suggests. This is the cheapest lever in the whole toolbox: it’s a test-time trick you can bolt onto any existing prompt pipeline this afternoon.

Problem & Motivation

LLMs generate text left-to-right and commit to it. Humans don’t write that way — you draft an email, wince at “Send me the data ASAP,” and rewrite it as “could you send this over when you get a chance?” That revision step is where a lot of quality comes from, and LLMs skip it entirely in standard one-shot prompting.

Before this paper, getting a model to improve its own output meant one of two expensive paths:

• Train a dedicated refiner on domain-specific (draft, feedback, revision) triples — works, but needs labeled data and a new model per task (PEER, CodeRL, Self-correction).
• Use RL against a reward model (RLHF-style) — works, but needs a trained reward model, policy-gradient training, and updates to the base model’s weights. Expensive to build, expensive to iterate on, and locked to whatever the reward model was trained to value.

Both approaches assume you need supervision — labeled examples of “bad output → good output” or a scalar reward function — to teach a model to improve itself. The paper’s bet: modern instruction-following LLMs (GPT-3.5/GPT-4 class) already know enough, in-context, to critique and revise text without any of that. The gap wasn’t capability, it was that nobody had asked the model to use the same capability twice in a row on its own output.

What’s New (Core Contribution)

• One model, three roles, zero training. Before: refinement needed either a separately trained refiner model or an RL loop with a reward model. Now: the same frozen LLM, called three times with three different few-shot prompts (generate / feedback / refine), does the whole job. No gradient updates anywhere.
• Feedback as a first-class, demonstrated skill. Before: prior “self-critique” work (Reflexion, Re3, Augmenter) prompted a single round of feedback for one task. Now: the paper shows feedback quality is the bottleneck (not the rewriting step) and demonstrates how to prompt for feedback that’s actionable (names a fix) and specific (names what to fix), with an ablation proving generic feedback (“improve the code”) underperforms specific feedback by a wide margin.
• History-conditioned iteration, not single-shot repair. Before: most self-critique setups do one critique-and-fix pass. Now: each refine step is conditioned on the entire trajectory of past drafts and feedback (y0, fb0, ..., yt, fbt), not just the latest pair — letting the model avoid repeating mistakes it already got called out on.
• Breadth of evidence. Before: self-critique demos existed for narrow tasks (story generation, code). Now: one algorithm, unmodified in structure, validated across 7 genuinely different task types (dialogue, code x2, math, sentiment, constrained generation, acronym generation) and 3+ model families — evidence this is a general property of strong LLMs, not a task-specific trick.

How It Works (Technically)

The whole method is one model M and three prompts: p_gen (generate), p_fb (feedback), p_refine (refine). Each prompt is a standard few-shot prompt — a handful of (input, output)-style example pairs prepended to the real input, the same in-context-learning mechanism GPT-3 popularized. No new machinery, just three different “personas” for the same model, switched by which prompt you use.

Step 1 — Initial generation. Given input x:

y0 = M(p_gen ∥ x)

∥ just means “concatenate into one prompt.” M produces a first-draft output y0, same as any normal LLM call.

Step 2 — Feedback. Feed the model its own output back in, with a different few-shot prompt whose examples are triples of (input, output, feedback):

fb_t = M(p_fb ∥ x ∥ y_t)

The few-shot examples teach the model what “good feedback” looks like — critically, actionable (says what to do) and specific (points at the exact phrase/line to change). “This code is slow because it uses six nested loops; use dynamic programming instead” is good feedback. “Improve the efficiency” is not — the paper measures this difference directly (see Results).

Step 3 — Refine. Feed the model the input, the entire history of drafts and feedback so far, and a third few-shot prompt whose examples are (input, draft, feedback, revised draft) quadruples:

y_{t+1} = M(p_refine ∥ x ∥ y0 ∥ fb0 ∥ ... ∥ y_t ∥ fb_t)

This is the detail that’s easy to miss: the refine call doesn’t just see the latest draft and latest critique — it sees the whole trail. That’s what lets the model avoid re-introducing a bug it already fixed two iterations ago.

Loop. Repeat Feedback → Refine until a stop condition fires: either a fixed iteration cap (paper uses up to 4), or the model itself emits a stop signal as part of its feedback (e.g., a numeric “quality score” that’s stopped improving). Return the last draft.

One input → one output, traced (Code Optimization, from the paper’s Figure 5):

1. x = “write a function that computes the cheapest way to make amount using coins [200, 300] costing [380, 550].”
2. y0 (generate) = a brute-force solution with 4-6 nested loops trying every combination — functionally correct, badly slow.
3. fb0 (feedback) = “This code is slow because it uses six nested loops to iterate through all possible combinations of coins… a more efficient approach would be [dynamic programming].” — specific (names the loops) and actionable (names the fix).
4. y1 (refine) = a ~6-line dynamic-programming solution, O(amount × coins) instead of exponential.
5. Stop condition fires (max iterations or model says “no further changes needed”) → return y1.

Architecture & data flow

flowchart TD
  X[Input x] --> GEN["M + p_gen<br/>(generate)"]
  GEN --> Y0[Draft y0]
  Y0 --> FB["M + p_fb<br/>(feedback)"]
  FB --> FBT["feedback fb_t<br/>(actionable + specific)"]
  FBT --> STOP{stop condition met?}
  STOP -- yes --> OUT[Return latest draft]
  STOP -- no --> REF["M + p_refine<br/>(refine, sees FULL history)"]
  REF --> YT1[New draft y_t+1]
  YT1 --> FB

The same model M is called three ways — as generator, critic, and rewriter — with quality (score) climbing and the gap between drafts shrinking each round. Click play to step through a run; watch how the feedback text changes as the draft converges.

The algorithm, simplified

# The one loop that is the whole paper. `llm(prompt) -> str` is a single model call.
def self_refine(x, llm, p_gen, p_fb, p_refine, max_iters=4):
    y = llm(p_gen + x)                     # step 1: initial draft
    history = [(y, None)]                  # keep every (draft, feedback) pair

    for t in range(max_iters):
        feedback = llm(p_fb + x + y)       # step 2: critique the CURRENT draft
        history[-1] = (y, feedback)

        if is_stop(feedback):              # model says "looks good" / score plateaued
            break

        # step 3: refine, conditioned on the FULL trajectory, not just the last pair
        context = x + "".join(f"{d}\n{fb}\n" for d, fb in history)
        y = llm(p_refine + context)
        history.append((y, None))

    return y

def is_stop(feedback_text):
    # task-specific: could parse a numeric score from feedback_text and
    # stop when it plateaus, or just check for a "no changes needed" phrase
    return "no further changes" in feedback_text.lower()

The only “trick” is that p_gen, p_fb, and p_refine are three different few-shot prompts — the model’s weights never change, only which persona it’s asked to play.

Built on Prior Work

Results & Evidence

Across GPT-3.5 (text-davinci-003), ChatGPT (gpt-3.5-turbo), and GPT-4, Self-Refine improved every task tested, by roughly 5-40 percentage points absolute:

Three findings matter more than the headline table:

1. Feedback quality is the whole game. An ablation (Table 2 in the paper) compares Self-Refine’s specific/actionable feedback against generic feedback (“improve the code”) and no feedback at all. Sentiment Reversal collapses from 43.2 → 31.2 → 0 as feedback gets vaguer; the task fails completely without feedback. This isn’t just “more compute helps” — the content of the critique is doing the work.
2. It’s not just “generate more and pick the best.” The authors compared Self-Refine’s single refined output against picking the best of k=4 independently sampled outputs (no feedback). Self-Refine still won in a 1-vs-k human preference test. Refinement-with-feedback beats brute-force resampling.
3. Weaker models can’t do this. Vicuna-13B could generate initial outputs fine but couldn’t reliably produce feedback in the required format, and even with hand-fed “oracle” feedback it often just repeated its draft or hallucinated instead of revising. Self-Refine requires a base model with strong instruction-following/few-shot ability — it is not a free lunch on smaller or weaker open models.

Caveats the paper is honest about: Math Reasoning barely moved (0.0-0.2%), because the model’s feedback step frequently just says “everything looks good” even when it isn’t (94% of ChatGPT feedback on wrong answers, per the paper) — the model can’t reliably detect its own math errors, so there’s nothing to act on. When an external correctness signal was substituted for self-detected errors, gains jumped 5+ points, telling you the ceiling here is bounded by self-detection ability, not by the refine step. All evaluation is English-only, and two of the “human-pref” metrics are validated against GPT-4-as-judge with only 68-82% correlation to actual human preference — real but imperfect agreement.

Real numbers from the paper's Figure 4 (averaged across ChatGPT, GPT-3.5, GPT-4): score by iteration for three tasks. Most of the gain lands in the first 1-2 rounds — the curve flattens fast, which is why a small fixed iteration cap (or an adaptive stop) is enough in practice.

How You’d Use It

This is the cheapest “critic” pattern available for a multi-agent or single-agent pipeline — it collapses the actor/critic split into one model and three prompts, no second agent, no reward model, no fine-tuning run. In your own build, it’s an afternoon of prompt-writing, not a training job. Concretely:

• As a quality gate on any generation step in an existing pipeline (report drafting, code generation, email/response generation) — wrap the existing generation call, add two more calls (feedback, refine), loop 2-3 times, done. This is a drop-in upgrade, not a new system.
• As a cheaper alternative to a second reviewer agent in your own multi-agent setup, when the risk of shared blind spots (the same model failing to see its own mistake) is acceptable — e.g., style/readability polish, not correctness-critical code.
• Where it will disappoint you: anything requiring the model to catch its own factual or logical errors (math, verifiable code correctness) — the paper’s own math result shows self-detection is the bottleneck, not refinement. Pair it with an external verifier (compiler, unit tests, calculator) wherever one exists — don’t treat “self-refine” as a substitute for actual verification.
• Where it needs a strong base model: don’t build this on a small local/open model without validating the feedback step first — Vicuna-13B failed outright. GPT-4/GPT-3.5-class or better is the tested floor.

Build Your Own (Minimal Recipe)

The smallest version that captures ~80% of the value:

1. Pick the task and write three few-shot prompts: p_gen (normal generation examples), p_fb (input/output/feedback triples where feedback is specific and actionable — the hard part), p_refine (input/output/feedback/revised-output quadruples).
2. Pick a stop condition: simplest is a fixed cap (2-3 iterations is often enough per the paper’s diminishing-returns curve — see the viz below). More sophisticated: have the feedback prompt emit a numeric quality score and stop when it plateaus.
3. Write the loop (see pseudocode above) — the one non-obvious part is passing the full history into the refine prompt, not just the latest draft/feedback pair.
4. The genuinely hard part: getting the feedback prompt to reliably produce specific, actionable critiques instead of vague praise or vague criticism. This is 90% prompt engineering — show the model, via your few-shot examples, exactly what “specific” looks like for your task (point at a line, a phrase, a clause — not “make it better”).
5. The second hard part: knowing when self-critique will and won’t work. If the task requires detecting a factual or logical error (not a stylistic one), plan to inject an external check (run the code, verify the math, check a fact) into the feedback step rather than trusting the model to self-detect.

Libraries: none beyond your existing LLM client (OpenAI/Anthropic SDK). No training framework, no vector DB, no new infra — this is a prompting pattern, not a system.

How to Improve It

1. Swap self-feedback for a second model or tool. The paper explicitly notes Self-Refine is “the only method that generates feedback using an LLM on its own output, for the purpose of refining with the same LLM” — test whether a different model (or a specialized critic model) as the feedback provider reduces shared blind spots, especially for correctness-sensitive tasks.
2. Ground feedback in an external verifier wherever one exists. The math result all but proves this: gains jump 5+ points when an outside source flags wrong answers. Wire in a compiler, unit-test runner, or calculator as a first-pass filter before asking the model to self-critique.
3. Learn a cheap stopping policy from the score trajectory instead of a fixed iteration cap — the paper shows steep diminishing returns after 1-2 iterations (see viz), so an adaptive stop could cut cost 30-50% with negligible quality loss.
4. Make it work on smaller/local models. Vicuna-13B failed zero-shot; a small amount of LoRA fine-tuning specifically on the feedback-and-refine format (not the task itself) might be enough to unlock Self-Refine-style gains on open-weight models without full supervised refiner training.
5. Combine with best-of-N. The paper tested Self-Refine (1 refined output) against k=4 unrefined samples and won — but never tested refining each of k samples and picking the best. That hybrid could push results further, at k× the cost.

Glossary

• Few-shot prompting / in-context learning — showing the model a handful of example input→output pairs in the prompt itself so it infers the pattern, with no weight updates.
• Instruction-following model — a model fine-tuned (e.g., via RLHF) to follow natural-language instructions well, as opposed to a raw base language model.
• RLHF (Reinforcement Learning from Human Feedback) — training a model’s weights using a reward signal derived from human preferences, typically via a separately trained reward model.
• Reward model — a model trained to score outputs by quality, used as the reward signal in RL-based tuning; Self-Refine avoids needing one.
• Greedy decoding / temperature — decoding settings controlling how deterministic (low temperature) vs. varied (high temperature) the model’s next-token choices are; the paper uses temperature 0.7.
• Ablation — an experiment that removes or degrades one component (here: feedback specificity, or the feedback step entirely) to measure its individual contribution.
• GPT-4-pref / human-pref — evaluation metrics where either GPT-4 or human raters do blind A/B comparisons between two candidate outputs to say which is better; used when there’s no automated task metric.
• Stop condition — the rule that ends the feedback-refine loop, either a fixed iteration count or a model-emitted signal that quality has plateaued.
• Constrained generation — a task requiring the output to include a large set of specific given keywords/concepts, used here as a stress test for self-refine (more concepts = more chances to miss one on the first try).