Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

TL;DR

Large language models were good at simple question-answering via few-shot prompting (show a handful of Q→A pairs, then ask a new question) but fell flat on anything requiring multiple reasoning steps, like word problems. This paper’s fix costs nothing to build: instead of writing “Q → A” exemplars, write “Q → reasoning → A” exemplars, where the reasoning is a short, human-style chain of intermediate steps. No training, no gradient updates, no new dataset — just a different way of writing eight examples in the prompt. Doing this made PaLM 540B jump from 18% to 57% solve rate on GSM8K math word problems, beating a fine-tuned GPT-3 with an external verifier, and it worked across arithmetic, commonsense, and symbolic reasoning benchmarks. The catch: this only works on large models (~100B+ parameters) — smaller models produce fluent-sounding but logically broken chains of thought and actually do worse than plain prompting.

Problem & Motivation

Two things were true in early 2022. First, scaling language models up kept improving them, but not on tasks needing multi-step reasoning — arithmetic and logic benchmarks stayed hard even for GPT-3-scale models (Rae et al., 2021). Second, there were two known ways to get a model to show its work: (1) train or fine-tune a model from scratch on a large dataset of hand-written rationales (Ling et al. 2017; Cobbe et al. 2021), which works but is expensive — good rationales are much harder and slower to produce than plain input→output labels; or (2) do standard few-shot prompting (Brown et al. 2020), which is cheap (a handful of examples, no training) but simply doesn’t work for reasoning tasks — the model jumps straight to an answer and gets multi-step problems wrong, and more model scale doesn’t reliably fix it.

The pain in one sentence: you could have cheap prompting or you could have reasoning ability, but not both.

What’s New (Core Contribution)

• Chain-of-thought (CoT) prompting. Before: few-shot exemplars were <input, output> pairs. Now: exemplars are <input, chain of thought, output> triples, where the chain of thought is a short natural-language walk-through of how a human would solve that specific problem. That’s the entire method — a change to how you write the prompt, not to the model.
• Empirical proof it works, at scale, across domains. The paper doesn’t just claim CoT helps; it shows large, consistent gains on 5 arithmetic benchmarks, 5 commonsense benchmarks, and 2 symbolic-reasoning tasks, across 5 different model families (GPT-3, LaMDA, PaLM, UL2, Codex).
• The discovery that CoT is an emergent ability of scale. Below ~100B parameters, CoT prompting doesn’t help and often hurts (small models write plausible-looking but logically wrong chains of thought). Above that threshold, the benefit appears and grows with scale. This reframes “model capability” — standard prompting numbers had been understating what large models can actually do.
• Careful ablations that rule out the boring explanations. The authors test whether the gain is just from “more output tokens = more compute” (no — a dot-filler placeholder doesn’t help) or just from “seeing an equation” (no — equation-only prompting barely helps on hard problems) or just from “priming knowledge recall” (no — putting the chain of thought after the answer doesn’t help). The gain specifically requires natural-language, step-by-step reasoning generated before the answer.
• Length generalization for symbolic tasks. With CoT, models generalize to symbolic problems (letter concatenation, coin-flip tracking) longer than anything seen in the few-shot exemplars — standard prompting cannot do this at all.

How It Works (Technically)

There is no new architecture and no training step here — the “mechanism” is entirely about what you put in the prompt and what that induces the model to generate. Walking one example through end to end (from Figure 1 of the paper):

1. Build a fixed prompt prefix of 4–8 exemplars. Each exemplar is a question, followed by 1–4 sentences of reasoning written in plain English, followed by “The answer is X.” Example: “Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. 5 + 6 = 11. The answer is 11.” These exemplars are written once, by hand, and reused for every test question in that dataset.
2. Append the real test question in the same “Q: … A:” format, but leave the answer blank.
3. Feed the whole thing to a frozen LLM and decode (the paper uses greedy decoding — always pick the highest-probability next token).
4. The model continues the pattern it was shown: because every exemplar had reasoning between “A:” and “The answer is,” the model generates its own reasoning steps for the new question before it commits to an answer. This is pure in-context pattern-matching — the model was never told “reason step by step”; it inferred that expectation from the shape of the exemplars.
5. Parse the final answer out of the generated text (e.g., by taking whatever follows “The answer is”).

Why does this help, mechanically? The paper’s own framing (Section 2) is: chain of thought lets the model allocate more computation (more forward passes / tokens) to harder sub-steps, gives an interpretable trace of how it got there, and is general — because it’s just language, it applies to any task a human could talk through, not only tasks with a formal symbolic representation (contrast with equation-based or program-based reasoning methods).

The paper also demystifies why it isn’t something simpler, via four controlled variants tested against standard and CoT prompting (Section 3.3, Figure 5):

That table is the real evidence for the mechanism: it’s specifically generating natural-language intermediate steps, in order, before answering that matters — not extra tokens, not equations, not knowledge activation alone.

Architecture & data flow

flowchart TD
  subgraph PROMPT["Fixed prompt prefix (written once by hand, reused every time)"]
    E1["Exemplar 1\nQ -> reasoning steps -> 'The answer is ...'"]
    E2["Exemplar 2\nQ -> reasoning steps -> 'The answer is ...'"]
    E3["... 4-8 exemplars total"]
  end
  TQ["New test question Q"] --> CAT["Concatenate:\nexemplars + new Q + 'A:'"]
  PROMPT --> CAT
  CAT --> LLM["Frozen LLM, >=100B params\n(no fine-tuning, greedy decode)"]
  LLM --> COT["Model generates its own\nchain of thought"]
  COT --> PARSE["Parse text after\n'The answer is'"]
  PARSE --> ANS["Final answer"]

flowchart LR
  subgraph STD["Standard prompting"]
    Q1["Q"] --> M1["LLM"] --> A1["Answer\n(often wrong on multi-step problems)"]
  end
  subgraph COTP["Chain-of-thought prompting"]
    Q2["Q"] --> M2["LLM"] --> R2["Reasoning steps"] --> A2["Answer\n(right, most of the time, at scale)"]
  end

Recreated from Figure 4/7/8's pattern (schematic, not the paper's raw numbers): standard prompting's accuracy stays roughly flat as models get bigger, while chain-of-thought prompting's accuracy stays flat too — until a threshold around 100B parameters, where it breaks away sharply. This is what "emergent ability" looks like on a chart. Drag the slider to see how the gap only exists on one side of the threshold.

Step through how the same test question is answered by a standard prompt vs. a chain-of-thought prompt. Click "Step" to watch the model consume the fixed exemplars, then either jump straight to an answer or work through intermediate steps first.

The algorithm, simplified

This is a prompting method, so the “algorithm” is prompt construction plus answer parsing — not a training loop. Self-consistency (Wang et al. 2022a, cited as immediate follow-up work) is included as a one-line optional wrapper because it’s the natural next step reader will want.

# The entire "method" — no gradients, no fine-tuning.

def build_cot_prompt(exemplars, question):
    # exemplars: list of (question, chain_of_thought, answer) tuples, written by hand ONCE per task
    blocks = []
    for q, cot, ans in exemplars:
        blocks.append(f"Q: {q}\nA: {cot} The answer is {ans}.")
    blocks.append(f"Q: {question}\nA:")   # leave the answer blank for the model to fill in
    return "\n\n".join(blocks)

def solve(llm, exemplars, question):
    prompt = build_cot_prompt(exemplars, question)
    generation = llm(prompt, decoding="greedy")   # llm(prompt) -> str; frozen, >=100B params
    # the model free-generates reasoning steps because every exemplar had them
    if "the answer is" in generation.lower():
        answer = generation.lower().split("the answer is")[-1]
    else:
        answer = generation   # fallback: couldn't parse, treat whole output as the answer
    return answer.strip(" .")

# Optional next step the paper points to (Wang et al. 2022a): self-consistency.
# Sample N chains of thought instead of one greedy decode, then majority-vote the final answers.
def solve_self_consistent(llm, exemplars, question, n=10):
    prompt = build_cot_prompt(exemplars, question)
    answers = [parse_answer(llm(prompt, decoding="sample", temperature=0.7)) for _ in range(n)]
    return most_common(answers)   # majority vote over N independent reasoning paths

Built on Prior Work

Results & Evidence

Headline numbers. PaLM 540B + CoT prompting: GSM8K math word problems jump from 18% (standard prompting) to 57% (CoT) solve rate — new state of the art, beating fine-tuned GPT-3 175B with an added verifier model (Cobbe et al. 2021). Similar new state-of-the-art results on SVAMP and MAWPS; within ~2% of SOTA on AQuA and ASDiv. On commonsense: StrategyQA improves to 75.6% (vs. prior best 69.4%), and on Sports Understanding the model beats an unaided human sports enthusiast (95.4% vs. 84%). On symbolic tasks (last-letter concatenation, coin-flip tracking), CoT + PaLM 540B reaches near-100% in-domain and, critically, generalizes to longer sequences than any exemplar shown (out-of-domain), while standard prompting cannot do this at all.

The scale threshold is real and consistent. Across GPT-3, LaMDA, and PaLM, CoT prompting is flat-to-negative below ~10–100B parameters and turns sharply positive above it, on every benchmark category tested (arithmetic, commonsense, symbolic). Smaller models generate fluent chains of thought that are logically wrong — this isn’t a formatting failure, it’s a reasoning failure.

Error analysis backs up the mechanism claim. Manually reading 50 LaMDA 137B chains of thought that reached correct answers: 48/50 were also logically and mathematically sound (not just coincidentally right). Of 50 that reached wrong answers, 46% were “almost correct” (one missing step, a calculator slip, a symbol-mapping error) and 54% had deeper semantic/coherence failures. A parallel comparison of PaLM 62B vs. 540B shows scaling fixes a large fraction of both error types.

Robustness checks. Three different human annotators independently wrote chains of thought for the same exemplars; all beat standard prompting by a wide margin despite stylistic variance. Exemplars sampled straight from the GSM8K training set (not hand-crafted at all) performed comparably. The method is also reported (Appendix, not detailed here) as robust to exemplar order and count.

What the evidence does NOT establish:

• No guarantee of faithful or correct reasoning. A chain of thought can be fluent and still be wrong, or (rarely) wrong-but-lucky. This paper does not verify that the stated reasoning is what actually drove the model’s output — it’s a plausible narrative, not a causal trace.
• Decoding was greedy for almost everything. Most results use a single greedy decode; there’s no exploration of the answer distribution except in the annotator-robustness check. (The authors themselves point to self-consistency — sampling many chains and voting — as the natural next step, done in follow-up work.)
• Cost at scale is a real limitation the authors flag themselves. The benefit requires ~100B+ parameter models, which is expensive to serve. The paper does not solve or test how to get this behavior into smaller, cheaper models.
• Manually written exemplars. For most benchmarks, the 4–8 chain-of-thought exemplars were hand-authored by the paper’s own authors — a small, uncontrolled source of task-specific tuning, even though the robustness study suggests the method isn’t overly sensitive to exact wording.
• No mechanistic account. The paper explicitly says it doesn’t answer why scale unlocks this ability, or whether the model is “actually reasoning” — it’s an empirical demonstration, not a theory.

How You’d Use It

This is the cheapest, highest-leverage lever in the whole prompting toolbox, and it underlies a large share of what “agentic” LLM systems do today:

• Any agent step that requires multi-step judgment — tool selection, task decomposition, plan generation — benefits from writing “think before you answer” into the few-shot exemplars (or, per the obvious zero-shot extension that followed this paper, just appending “Let’s think step by step”). This is why almost every production agent prompt you write should have a reasoning section before the action/tool-call section.
• Debuggability you get for free, in your own harness. Because the reasoning is generated in natural language before the answer, it’s a free interpretability layer — you can read why your agent did what it did straight out of the trace, and eyeball-audit failures instead of treating the model as a black box. That’s worth wiring into your logging/observability layer, not just leaving in the raw completion.
• It’s the ancestor of ReAct, Tree-of-Thought, and self-consistency. If your own agent harness already has tool-calling loops with a “thought” field, you’re running a direct descendant of this paper. Knowing that lineage tells you why a prompting change (“add explicit reasoning steps to your few-shot examples or system prompt”) will measurably move accuracy in your own stack — you’re not guessing, you’re applying a documented, benchmarked technique.
• A cheap first diagnostic when your own LLM feature underperforms on multi-step tasks. Before reaching for fine-tuning or a bigger model, check whether your prompt is even asking for a reasoning trace. This paper is the evidence that the answer is often “no, and that alone explains a lot of the accuracy gap” — and the fix costs an afternoon of prompt writing, not a training run.
• Caveat to carry into your own deployment decisions: this only reliably works on large, capable models (broadly, today’s frontier-class models, not small/local ones) — factor that in before betting on CoT-style gains from a cost-optimized small model in your own stack.

Build Your Own (Minimal Recipe)

You can reproduce ~80% of this paper’s value in under an hour with no training:

1. Pick 4–8 representative examples of the task (skew toward examples that need 2+ reasoning steps — CoT’s benefit scales with problem difficulty, per the SingleOp vs. GSM8K contrast in the paper).
2. Hand-write a short reasoning trace for each, 1–4 sentences, in the style a person would actually think through the problem, ending in a consistent terminal phrase like “The answer is X.” Consistency of that terminal phrase is what makes answer-parsing reliable.
3. Concatenate exemplars + the real question into one prompt, leaving the final answer blank (see build_cot_prompt above).
4. Use a large, capable model and greedy or low-temperature decoding for a first pass.
5. Parse the answer with a simple string split on your terminal phrase; keep the full generation around for debugging/audit.
6. The one genuinely hard part: exemplar quality. Bad or inconsistent reasoning style in your hand-written exemplars will propagate into the model’s outputs (garbage in, garbage out) — this is prompt engineering, and it benefits from the same iterate-and-eyeball process as any other prompt.
7. The other hard part, if you scale this to production: answer-parsing robustness. Real generations don’t always contain your terminal phrase cleanly — budget for a regex/fallback parser and a small eval set to catch silent parsing failures, not just reasoning failures.
8. Nice-to-have upgrade: wrap it in self-consistency (solve_self_consistent above) — sample N chains of thought instead of one, majority-vote the final answer. Costs N× the inference calls, buys meaningfully higher accuracy; this is literally the next paper Wang et al. wrote off the back of this one.

Libraries/models to reach for: any current frontier-class chat/completion model via its API (temperature=0 for a first pass, temperature≈0.5–0.8 + majority vote for self-consistency); no fine-tuning infra needed.

How to Improve It

• Zero-shot CoT. You don’t even need hand-written exemplars — appending “Let’s think step by step” to the question alone recovers much of the benefit (Kojima et al., 2022, immediate follow-up). Worth A/B-testing against few-shot CoT before paying the cost of writing exemplars.
• Self-consistency / majority voting. Sample multiple chains of thought at nonzero temperature and vote on the final answer instead of trusting one greedy decode (Wang et al., 2022a) — directly addresses the “no guarantee of correct reasoning” caveat above by treating each chain as one noisy vote rather than ground truth.
• Verify or rerank the chain, not just the answer. The paper shows ~46% of wrong answers were “almost correct” — a lightweight verifier step (or a second LLM call asking “check this reasoning for errors”) could catch and fix a meaningful chunk of failures cheaply, without touching the base model.
• Distill the capability into a smaller model. Since CoT only “turns on” above ~100B parameters, generate CoT traces with a large model and fine-tune a small model on them (this is close to what Cobbe et al. did, and to later work like STaR, Zelikman et al. 2022) — trades a one-time training cost for cheap inference that keeps most of the reasoning benefit.
• Automate or optimize exemplar selection. The paper’s exemplars were hand-picked once and reused; a system that selects (or retrieves) the most relevant CoT exemplars per test question, or that automatically searches over exemplar phrasing, is a direct, testable extension — and it’s the kind of “prompt optimization” layer you could productize.

Glossary

• Few-shot prompting — showing a language model a handful of example input→output pairs in the prompt itself (no training) before asking it to handle a new input.
• Exemplar — one example in that handful; here, a <question, reasoning, answer> triple instead of just <question, answer>.
• Chain of thought (CoT) — the natural-language sequence of intermediate reasoning steps a model generates between the question and the final answer.
• Greedy decoding — generating text by always picking the single highest-probability next token, rather than sampling; deterministic, one output per prompt.
• Solve rate — accuracy: percent of test questions the model got right.
• Emergent ability — a capability that is near-zero across a range of smaller model sizes and then appears sharply once scale crosses some threshold, rather than improving smoothly with scale.
• In-domain vs. out-of-domain (OOD) generalization — here, whether test problems need the same number of reasoning steps as the exemplars (in-domain) or more steps than any exemplar showed (OOD/length generalization).
• GSM8K — a benchmark of grade-school math word problems (Cobbe et al., 2021), used as this paper’s headline result.
• Self-consistency — a follow-up technique (Wang et al., 2022a): sample many chains of thought instead of one, then take the majority-vote final answer.
• Ablation — an experiment that removes or swaps one piece of a method to isolate which part is actually responsible for its effect (used here to rule out “more tokens” and “equation-only” as the real explanation for CoT’s benefit).