Language Models are Few-Shot Learners

TL;DR

Fine-tuning a pretrained language model still requires thousands of labeled examples for every new task, which doesn’t scale and doesn’t match how humans learn. This paper trains GPT-3, a 175-billion-parameter autoregressive transformer — 10x bigger than any prior dense language model — and tests it purely as a frozen, in-context learner: give it a task description and a handful of examples inside the prompt, take one forward pass, and read off the answer. Few-shot performance turns out to improve smoothly and often dramatically with model scale, and the gap between zero-shot, one-shot, and few-shot performance widens as models get bigger — meaning larger models aren’t just better next-token predictors, they’re better at picking up a task from a few clues. GPT-3’s few-shot results are competitive with (and on some benchmarks beat) fine-tuned state-of-the-art systems, and it can write ~500-word news articles that human readers can only distinguish from human-written ones at ~52% accuracy — essentially chance.

Problem & Motivation

The dominant NLP recipe by 2020 was: pretrain a big transformer on unlabeled text, then fine-tune it on a labeled dataset for whatever task you care about. That second step is the bottleneck. It typically needs thousands to hundreds of thousands of task-specific labeled examples, which is expensive to collect and has to be redone for every new task. Worse, a model fine-tuned hard on one narrow dataset can learn spurious correlations specific to that dataset rather than the underlying task, so a benchmark score that looks “human-level” can overstate real-world competence. Humans don’t work this way — a one-line instruction or two or three examples is usually enough for a person to pick up a new task. Prior attempts at getting language models to do the same thing via prompting alone (“in-context learning,” first shown informally in GPT-2) were real but weak: GPT-2 scored only 4% on Natural Questions in this setting, tens of points behind fine-tuned baselines. The open question this paper asks: does that gap close if you scale the model itself by another order of magnitude or two?

What’s New (Core Contribution)

• Before: the largest dense language models topped out around 17B parameters (Turing-NLG). Now: GPT-3 at 175B parameters, roughly 10x larger than anything trained before it, with 7 smaller siblings (125M to 13B) trained the same way for controlled comparison.
• Before: in-context learning (prompting a frozen LM with examples instead of fine-tuning) was a side observation in the GPT-2 paper. Now: a systematic study of zero-shot, one-shot, and few-shot performance across more than two dozen standard NLP benchmarks plus several newly designed synthetic tasks (arithmetic, word unscrambling, using a novel word after one definition) built specifically to probe rapid task adaptation.
• Before: nobody had shown whether downstream task performance (not just validation loss) follows the same smooth power-law scaling with model size that Kaplan et al. (2020) found for loss. Now: this paper shows it largely does — and further shows the few-shot advantage over zero-shot itself grows with scale, evidence that scale specifically improves in-context (meta-)learning, not just raw language modeling.
• Before: contamination of test sets by pretraining corpora scraped from the web was a known but rarely quantified risk. Now: the paper builds explicit tooling to measure train/test overlap in Common Crawl-scale data and reports its measured (and honestly, its missed) impact on results.

How It Works (Technically)

GPT-3’s architecture is not the novelty — it’s essentially the GPT-2 decoder-only transformer (pre-normalization, the same reversible byte-pair-encoding tokenizer), with one tweak borrowed from the Sparse Transformer: attention layers alternate between standard dense attention and locally-banded sparse attention, which keeps compute manageable at extreme scale. All models use a 2048-token context window. Table 2.1 in the paper lists all 8 trained sizes; GPT-3 has 96 layers, model width 12,288, and 96 attention heads.

Training data is a filtered, deduplicated mixture: Common Crawl (570GB after filtering, filtered by similarity to known high-quality corpora), an expanded WebText2, two books corpora, and Wikipedia. Crucially, these aren’t sampled proportional to their raw size — cleaner corpora (Wikipedia, WebText2) are oversampled (seen up to 3.4 times across the 300-billion-token training run) while noisy Common Crawl is undersampled (seen less than once). Training itself is a completely standard next-token cross-entropy objective — nothing new happens here either.

The 8 model sizes actually trained (Table 2.1), spheres scaled so volume tracks parameter count. Orange is GPT-3 175B — drag to orbit and see just how large the jump from 125M to 175B really is (about 1,400x).

The actual contribution is the evaluation protocol, and it’s worth tracing end to end. For any task, at inference time you build one text prompt out of three pieces: (1) an optional natural-language task description, (2) K worked examples pulled from that task’s own training set, each shown as context → completion, and (3) the new context you actually want answered. That whole string is fed through the frozen model in a single forward pass — no gradient ever touches the weights. If the task is multiple-choice or classification, you don’t even generate text: you compare the model’s log-probability of each candidate completion and pick the highest-scoring one. If it’s free-form (translation, open QA), you decode with beam search (beam width 4). K ranges from 0 (zero-shot: instruction only) up to however many examples fit in the 2048-token window — typically 10 to 100 (few-shot). One-shot (K=1) is treated as a separate condition because it best matches how a task is often explained to a human worker: one example, not zero, not dozens.

Concrete trace — English→French translation, few-shot: prompt = "Translate English to French:" + 10 pairs like "the cat sat on the mat" → "le chat était assis sur le tapis" (each pulled from a small parallel corpus) + the new English sentence you actually want translated. The model reads all of that as context and its next tokens are the French translation, scored against a reference with BLEU. Nothing about the model’s weights has changed since pretraining finished; the “learning” of this specific task happened entirely inside that one forward pass, conditioned on the 10 examples sitting in the prompt.

The paper’s framing (Figure 1.1) splits this into an outer loop and an inner loop: the outer loop is pretraining itself — gradient descent over hundreds of billions of tokens, where the model absorbs a huge breadth of latent skills and pattern types. The inner loop is a single forward pass at inference time, where the model recognizes which of those latent skills the prompt is asking for and applies it — this inner loop is what they call in-context learning, and the whole two-level structure is what they call meta-learning. The paper is careful to flag that this is a name for the behavior, not a claim about the underlying mechanism — whether the model is truly learning a new task “from scratch” inside that forward pass, or just recognizing a task pattern it already absorbed during pretraining, is explicitly left open (see Limitations).

There’s no new math to demystify beyond one relationship carried over from prior work: Kaplan et al. 2020 showed that pretraining loss falls as a smooth power law in model size and compute — plotted on log-log axes it’s a straight line. This paper’s headline empirical finding is that the same smooth trend, extended two more orders of magnitude, keeps holding for validation loss (Figure 3.1), and — more importantly — a similar smooth trend shows up in aggregate downstream task accuracy (Figure 1.3), for zero-, one-, and few-shot alike, with few-shot rising fastest.

Architecture & data flow

flowchart TD
  subgraph Pretraining["Outer loop: pretraining (done once)"]
    C["Common Crawl + WebText2 + Books1/2 + Wikipedia<br/>(quality-weighted sampling, 300B tokens)"] --> T["GPT-3 transformer<br/>(125M to 175B params, 2048-token context)"]
    T -->|"next-token cross-entropy"| T
  end
  subgraph Inference["Inner loop: in-context learning (every task, every query)"]
    D["Task description"] --> P["One prompt string"]
    K["K worked examples<br/>(context -> completion), K = 0 to ~100"] --> P
    Q["New query"] --> P
    P --> F["Frozen forward pass<br/>0 gradient updates"]
    F --> O["Completion<br/>(scored completion, or beam-search text)"]
  end
  T -.frozen weights.-> F

The prompt is built once per query — task description, then K demonstrations, then the new case — and pushed through the frozen model in a single forward pass. K slides from 0 (zero-shot) up toward the context-window limit; nothing here ever updates a weight.

The four settings, compared

flowchart LR
  FT["Fine-Tuning<br/>1,000s+ labeled examples<br/>weights updated"] --> FS["Few-Shot<br/>10-100 examples in the prompt<br/>0 weight updates"]
  FS --> OS["One-Shot<br/>1 example in the prompt"]
  OS --> ZS["Zero-Shot<br/>instruction only, 0 examples"]

The evaluation protocol, simplified

def few_shot_prompt(task_description, train_examples, query, k):
    # K demonstrations drawn from the task's own train set, K bounded by the 2048-token context window
    demos = random.sample(train_examples, k)
    prompt = f"{task_description}\n\n" if task_description else ""
    for context, completion in demos:
        prompt += f"{context}\n{completion}\n\n"      # show k worked examples, context -> completion
    prompt += f"{query}\n"                             # the one case we actually want answered
    return prompt

def evaluate(model, task_description, train_examples, test_examples, k, candidates=None):
    # model is FROZEN here: no gradient step happens anywhere in this function
    scores = []
    for query, gold in test_examples:
        prompt = few_shot_prompt(task_description, train_examples, query, k)
        if candidates:                                  # multiple-choice / classification tasks
            # pick whichever candidate the frozen model scores most likely, length-normalized
            best = max(candidates, key=lambda c: model.logprob(prompt + c) / len(c))
            scores.append(best == gold)
        else:                                            # free-form generation (translation, open QA)
            completion = model.generate(prompt, beam_width=4, length_penalty=0.6)
            scores.append(score_fn(completion, gold))     # F1 / BLEU / exact match, task-dependent
    return mean(scores)

Built on Prior Work

Results & Evidence

The headline finding is scaling behavior, not any single benchmark number: validation loss keeps following Kaplan et al.’s power law for two more orders of magnitude with only minor deviation (Figure 3.1), and aggregate accuracy across 42 accuracy-scored benchmarks rises with scale too — with the few-shot curve climbing faster than zero-shot as models get bigger (Figure 1.3), meaning larger models get disproportionately more value out of a handful of examples, not just from more parameters per se.

Schematic reproduction of the qualitative trend in Figure 1.3 (not digitized paper data): few-shot accuracy pulls further ahead of zero-shot as model size increases across the 8 trained sizes.

Selected task results (GPT-3 175B unless noted):

• PTB language modeling: 20.5 perplexity zero-shot, a new SOTA by 15 points over the prior 35.8.
• LAMBADA (predict the last word of a long passage): 76.2% accuracy zero-shot (already beats the prior 68.0% SOTA) → 86.4% few-shot.
• CoQA (conversational QA): 81.5 F1 zero-shot → 84.0 one-shot → 85.0 few-shot, closing in on fine-tuned SOTA.
• TriviaQA (closed-book, answer from parameters only, no retrieval): 64.3% zero-shot → 71.2% few-shot — the few-shot number beats the fine-tuned SOTA in the same closed-book setting.
• Arithmetic (novel synthetic task): near-perfect 2-digit addition/subtraction, strong 3-digit performance (94.2% on 3-digit subtraction), degrading on 4-5 digit and multiplication — and the authors spot-checked that specific 3-digit problems don’t appear verbatim in the training data, arguing against pure memorization.
• Synthetic news articles: human raters distinguishing GPT-3-175B-written ~500-word articles from human-written ones scored only ~52% accuracy (chance is 50%); smaller control models were much easier for humans to catch.

Caveats the paper itself surfaces, which matter as much as the wins:

• Some task types barely move with scale at all — comparison-style tasks (WIC: are two word senses the same; ANLI: does one sentence imply another) and some reading-comprehension sets (RACE, QuAC) stay near-chance even at 175B few-shot, which the authors partly attribute to using a purely autoregressive (not bidirectional) architecture.
• A bug in their contamination filter let some benchmark test data leak into the Common Crawl-derived training set; they measured the effect as small for most datasets but withheld or asterisked results where it looked significant, rather than pretending it away.
• It’s explicitly unresolved whether few-shot performance reflects genuine on-the-fly learning or pattern-recognition against something already absorbed during pretraining — the paper states this as an open question, not a solved one.
• No fine-tuned GPT-3 baseline was run at all (by design, to isolate task-agnostic performance) — so there’s no measurement of how much headroom remains above the reported few-shot numbers.
• 175B parameters is expensive to run inference on; every result in the paper reflects a model that is, in the authors’ own words, “inconvenient” to deploy at this size.

How You’d Use It

This paper is the origin story for prompt-engineering as a real substitute for fine-tuning, and its evaluation protocol (§2.4) is close to a checklist you’re probably already running informally: try a natural-language instruction, add a handful of worked examples, tune K on a small dev set before locking in a production prompt, and for classification/extraction tasks score candidate completions by log-probability rather than parsing free text — it’s more stable.

The direct build-vs-buy argument, for your own applications and automations: when a task doesn’t have (or can’t legally use) a large labeled dataset — which is most tasks in an early build — few-shot prompting a large frozen model beats fine-tuning a small one. No training pipeline, no per-task retraining ops burden, and iteration speed measured in prompt edits rather than retraining runs. The paper’s own evidence is also the caveat: the in-context learning curve (more examples helping more) barely exists at small model scale — this is a frontier/large-model behavior, not something you get for free out of a small local model, so route ambiguous, low-data, or rapidly-changing-spec tasks in your pipeline to a large hosted model rather than a fine-tuned small one, and reserve fine-tuning for the tasks where you do have volume and stability.

Build Your Own (Minimal Recipe)

You won’t train a 175B model, and you don’t need to — the 80%-value version is a small harness that reproduces the behavior this paper documents, on top of any existing large model you can call via API or run locally:

1. Pick a task with a clean train/test split and an automatic metric (sentiment classification, a small extraction task, simple QA).
2. Build the prompt template exactly as in few_shot_prompt above: task description, then K (context, completion) pairs sampled from train, then the held-out query.
3. Sweep K = 0, 1, 4, 16 (or however many fit your context budget) and plot accuracy vs. K on a small held-out set — this reproduces the paper’s “in-context learning curve” (their Figure 1.2) for your own task.
4. Run the identical harness against two or three model sizes (a small local model, a mid-size one, a frontier one) to see the scale-dependence yourself — this is the paper’s central claim, and it’s cheap to check.
5. For classification/multiple-choice tasks, score by comparing log-probability of each candidate completion rather than free generation, as in §2.4 — fewer moving parts, more stable numbers.

The genuinely hard parts: (a) prompt formatting — delimiters, example order, and even whitespace measurably swing results, so budget for a small template grid-search per task rather than trusting one “obviously right” format; (b) which K examples you pick matters more than intuition suggests — near-duplicate or unrepresentative demonstrations actively hurt; (c) you need an honest zero-shot/majority-class baseline to know whether K>0 is actually buying you anything, or just adding noise. No training code is required anywhere in this recipe — it’s pure prompting infrastructure, buildable against any hosted LLM API or a local server (vLLM/Ollama) in an afternoon.

How to Improve It

1. Bidirectionality for comparison tasks. The paper’s own limitations section flags autoregressive-only architecture as a likely reason WIC/ANLI/RACE-style “compare two things” tasks stay weak even at scale — worth testing whether a modern bidirectional-pretrained model with few-shot capability closes that specific gap, rather than assuming scale alone will.
2. Distillation. The paper explicitly names distilling a giant few-shot-capable model down to a small task-specific model as untried at this scale in 2020 — today’s teacher/student distillation pipelines from frontier models are exactly this pattern; worth benchmarking how much few-shot quality actually survives distillation for a specific client task, rather than assuming it’s lossless.
3. A reusable contamination checker. Their own train/test overlap detector shipped with a bug that let some leakage through. A small, cheap n-gram-overlap tool that checks a benchmark’s eval set against a training corpus (or its dedup manifest) before you trust any LLM eval result is a directly buildable, reusable artifact — not paper-specific.
4. Retrieval instead of closed-book. TriviaQA and other QA results here are deliberately closed-book (answer from parameters only, nothing retrieved). Pairing the same few-shot protocol with retrieval — put relevant passages in the context alongside the demonstrations — is a straightforward extension that current RAG practice has already validated; worth quantifying the lift on the paper’s own closed-book numbers specifically.
5. Calibration. The paper flags that GPT-3 isn’t well-calibrated on novel inputs (higher variance than humans on the same benchmarks). A testable add-on: apply a calibration step (e.g., contextual/temperature calibration) on top of the identical few-shot harness and measure whether it reduces sensitivity to prompt/example ordering — a cheap, measurable win if it works.

Glossary

• Autoregressive language model — a model that predicts the next token from everything before it, one token at a time.
• Fine-tuning — updating a pretrained model’s weights on a labeled dataset built for one specific task.
• In-context learning — the model “picks up” a task purely from instructions/examples placed in its input text, with zero weight updates.
• Zero-/one-/few-shot — 0, 1, or many (typically 10–100) task examples shown in the prompt before the real query.
• Meta-learning (as used in this paper) — the outer/inner-loop framing: pretraining (outer loop) teaches broad task-recognition skill; a single forward pass at inference (inner loop) applies it to whatever the prompt asks for.
• Perplexity — a measure of how well a language model predicts text; lower means the model was less “surprised,” on average, by the true next tokens.
• Power law — a relationship where one quantity scales as a fixed exponent of another; on log-log axes it plots as a straight line (here: loss/accuracy vs. model size or compute).
• Cross-entropy loss — the training objective; measures how far the model’s predicted next-token probabilities are from the actual next token.
• BPE (byte-pair encoding) — the sub-word tokenization scheme that turns raw text into the units the model actually operates on.
• Sparse attention — an attention pattern where each token attends to only a subset of prior tokens (alternated here with full/dense attention layers) to save compute at long context lengths.
• Common Crawl — a large, continuously updated, noisy scrape of the public web, used as a pretraining data source and filtered/deduplicated before use here.
• Data contamination — when benchmark test examples leak into training data, artificially inflating measured performance.
• Beam search — a decoding strategy that keeps several likely partial sequences at once and expands the best ones; used here for free-form generation tasks like translation.
• SuperGLUE / LAMBADA / TriviaQA / CoQA / ANLI / WIC / RACE / QuAC — standard NLP benchmark datasets/suites covering language understanding, question answering, and reasoning, used across the paper to test GPT-3 on many different skills at once.