Foundations of Large Language Models

TL;DR

This is not a single-result research paper; it is a foundations textbook that distills the four things you need to understand to reason competently about any LLM: (1) pre-training (how a model absorbs world knowledge from raw text via self-supervised objectives), (2) generative modeling at scale (the decoder-only Transformer, scaling laws, and the engineering to train and serve long contexts), (3) prompting (in-context learning, chain-of-thought, decomposition, self-refinement), and (4) alignment (instruction tuning, RLHF, and DPO) plus (5) inference (decoding algorithms, acceleration like speculative decoding, and inference-time scaling à la o1/R1).

The key insight the book keeps returning to: a single model trained to predict the next token, at sufficient scale, becomes a universal problem solver that you then steer with fine-tuning, alignment, and prompting rather than retraining from scratch. The headline “result” is conceptual — it gives you the load-bearing equations (QKV attention, the language-modeling loss, power-law scaling, the RLHF objective, the DPO derivation, the speculative-decoding acceptance rule) and the intuition to use them. If you build or sell AI systems, this is the mental model that lets you judge which knob (data, parameters, prompt, RL, decoding) to turn for a given client problem.

Problem & Motivation

Before LLMs, the dominant NLP recipe was: pick a task, collect a large labeled dataset for that task, train a specialized model from scratch. This was expensive, brittle, and didn’t transfer — a sentiment classifier knew nothing about translation. Every new client problem meant a new labeling effort and a new model.

The pain the book frames: how do you get one model that handles diverse problems without task-specific supervised data for each? The answer that reorganized the field is large-scale self-supervised pre-training — train on the structure of raw text itself (predict the next token, or fill in masked tokens), which requires no human labels and exists in near-infinite supply. The model picks up syntax, facts, and reasoning patterns as a side effect. You then adapt this foundation model cheaply via fine-tuning, alignment, or just prompting.

But that creates new problems the book systematically addresses: pre-training is enormously compute-hungry (needs scaling laws to budget it), the resulting base model is knowledgeable but not helpful or safe (needs alignment), it doesn’t automatically reason through hard problems (needs prompting strategies and inference-time scaling), and serving it is slow and memory-bound (needs decoding tricks like KV-caching and speculative decoding). Each chapter is one of these downstream problems.

What’s New (Core Contribution)

This is a synthesis/textbook, so “novelty” means clarity and coverage rather than a new algorithm. Its genuine contributions as a reference:

• A clean five-stage mental model of the LLM lifecycle. Before: the techniques (BERT, GPT, RLHF, CoT, o1) are scattered across hundreds of papers with inconsistent notation. Now: one consistent notation (e.g., framing the LLM uniformly as a policy π(a|s) = Pr(y_t | x, y_<t)) ties pre-training, prompting, RL alignment, and inference together so the connections are visible.
• Derivations, not just descriptions. Most surveys say “DPO removes the reward model.” This book actually derives the DPO objective from the RLHF objective step by step, so you see exactly which assumption (fixed reward + reference model) makes the simplification work. Same for the policy-gradient / REINFORCE result and the speculative-decoding acceptance probability.
• Explicit separation of training-time vs. inference-time scaling. It frames o1/R1-style “long thinking” as a third scaling axis (after data and parameters) — inference-time compute — and organizes the zoo of test-time methods (context scaling, search scaling, output ensembling, generate-and-verify) into a usable taxonomy.
• Engineering reality, not just theory. Chapter 2 covers distributed training, KV-cache, grouped/multi-query attention, and RoPE position interpolation — the systems concepts you actually hit in production, demystified.

How It Works (Technically)

The book has five chapters; the through-line is one next-token predictor, steered five different ways. I’ll trace the load-bearing mechanism of each.

1. Pre-training: learn from raw text with no labels

The core trick is self-supervised learning — manufacture supervision from the data itself. Two dominant flavors:

• Decoder-only / causal LM (GPT family): predict the next token given all previous tokens. Objective is to maximize log-likelihood Σ_i log Pr(x_i | x_0, ..., x_{i-1}). In plain English: nudge the model’s parameters so the actual next word in the corpus gets higher probability. That single objective, over trillions of tokens, is enough to learn grammar, facts, and reasoning.
• Encoder-only / masked LM (BERT): randomly hide ~15% of tokens and predict them from both sides. Great for understanding/classification, but not generation.

2. The decoder-only Transformer (the engine)

Input tokens become d-dimensional vectors e_i = token embedding + positional embedding. The body is L stacked blocks, each with two sub-layers: self-attention and a feed-forward network (FFN), wired with residual connections and layer norm (pre-norm LNorm(F(input)) + input is the modern default — it trains more stably at depth).

The heart is QKV attention:

Attention(Q, K, V) = Softmax( (Q Kᵀ / √d) + Mask ) V

What this does operationally: each token emits a query (what am I looking for?), every token exposes a key (what do I offer?) and a value (my actual content). Q Kᵀ is an m×m grid of dot-products — how much token i should attend to token k. Divide by √d to keep the numbers from blowing up (large d → large dot products → saturated softmax → vanishing gradients). Softmax turns each row into attention weights that sum to 1; multiply by V to get a weighted blend of all tokens’ content. The causal Mask sets entry (i,k) to −∞ when k > i, so a token can never attend to the future — that’s what makes generation left-to-right.

Multi-head attention splits the d dimensions into τ sub-spaces, runs attention independently in each (so different heads can specialize — one tracks syntax, another tracks coreference), then concatenates and projects back. On top of the last block, a softmax over the vocabulary Softmax(H_L W_o) produces the next-token distribution.

Schematic causal self-attention: each row is a query token, each column a key token. Brighter = more attention. The upper triangle is masked to zero (a token can't see the future). Click "step" to watch generation reveal one token at a time.

3. Scaling laws (how to budget the training)

Empirically, test loss falls as a power law in the variable you scale: L(x) = a·x^b. Kaplan et al. found, e.g., L(N) ≈ (N / 8.8e13)^{-0.076} for parameter count N and L(D) ≈ (D / 5.4e13)^{-0.095} for data size D. Plain English: doubling parameters or data gives a predictable, diminishing loss reduction — there’s a slow phase, a power-law phase, and a convergence phase (you hit irreducible error). This is enormously practical: you can fit the curve on small runs and predict the loss of a 10× bigger run before spending the GPU budget, and decide how to split a fixed compute budget between more parameters vs. more data. Scaling also produces emergent abilities — capabilities (e.g., multi-step arithmetic) that are absent in small models and appear past a size threshold.

4. Prompting & chain-of-thought (steer without training)

In-context learning: the model learns the task from examples in the prompt, no weight updates. Chain-of-thought (CoT) prompting tells the model to emit intermediate reasoning steps before the answer (“Let’s think step by step”). Mechanistically, generating reasoning tokens gives the model more serial computation and a scratchpad — each intermediate result becomes context the next step can condition on. The book also covers problem decomposition (break into sub-questions) and self-refinement (generate → critique → revise, looped).

5. Alignment: RLHF and DPO (make it helpful/safe)

The book frames alignment as RL: the LLM is a policy π(a|s), where state s = (x, y_<t) is the prompt-so-far, action a = y_t is the next token, and a reward model r(x, y) (trained on human preference rankings via a pairwise ranking loss) scores completed outputs.

RLHF (the PPO route): train a reward model from human comparisons, then fine-tune the policy to maximize reward minus a KL penalty that keeps it close to the original model (so it doesn’t reward-hack into gibberish). The policy-gradient intuition (REINFORCE): ∇J(θ) = E[ R(τ) · ∇ log π(τ) ] — push up the probability of trajectories that got high reward, push down low-reward ones. A learned baseline/value function is subtracted to reduce variance (advantage = reward − baseline), and PPO clips the update so each step doesn’t move the policy too far.

DPO is the elegant shortcut. It assumes the reward model and reference model are fixed, then algebraically folds the reward into the policy, showing the optimal policy is π*(y|x) ∝ π_ref(y|x)·exp(r(x,y)/β). Inverting that lets you express the reward in terms of the policy itself, so you can optimize directly on preference pairs (chosen y_w vs. rejected y_l) with a supervised-style loss — no reward model, no RL loop, no sampling. This is why DPO is the default for most teams: it gets ~80% of RLHF’s benefit with the operational simplicity of fine-tuning.

6. Inference: decoding, acceleration, and inference-time scaling

Generation is autoregressive and memory-bound. Two big ideas:

• Speculative decoding (free speedup, no quality loss): a small fast draft model guesses the next τ tokens; the big verification model checks all τ in one parallel forward pass. Accept a draft token ŷ with probability based on the ratio of the two models’ probabilities p(ŷ)/q(ŷ); on rejection, resample from the corrected distribution. Because verification is parallel, you generate multiple tokens per expensive forward pass while provably matching the big model’s output distribution.
• Inference-time scaling (the o1/R1 idea): spend more compute at test time instead of training a bigger model. Categories: context scaling (more/better context, RAG), search scaling (longer outputs, wider beams, tree/MCTS-style search over reasoning steps), output ensembling (self-consistency: sample many CoT paths, majority-vote), and generate-and-verify thinking paths (a verifier scores partial solutions, enabling backtracking and self-correction). Training-based variants fine-tune the model to produce long, self-correcting reasoning traces.

Architecture & data flow

flowchart TB
  subgraph PT[1. Pre-training]
    A[Raw text corpus] --> B[Self-supervised loss<br/>next-token / masked-token]
    B --> C[Base model<br/>knowledgeable, not helpful]
  end
  C --> D[2. Decoder-only Transformer<br/>L blocks: attention + FFN]
  D --> SL[Scaling laws<br/>budget params vs data]
  C --> E[4. Alignment]
  subgraph AL[4. Alignment]
    E --> F[Instruction / SFT]
    F --> G[RLHF: reward model + PPO]
    F --> H[DPO: direct on preference pairs]
  end
  G --> I[Aligned assistant]
  H --> I
  I --> J[3. Prompting<br/>in-context, CoT, decomposition]
  J --> K[5. Inference]
  subgraph INF[5. Inference]
    K --> L[Decoding + speculative decoding]
    K --> M[Inference-time scaling<br/>search / ensemble / verify]
  end
  M --> N[Final answer]
  L --> N

The algorithm, simplified

The single most reusable mechanism for the reader is DPO — alignment without an RL loop. Here is its core, with the paper’s names:

import torch, torch.nn.functional as F

def dpo_loss(policy, ref_policy, prompt, y_chosen, y_rejected, beta=0.1):
    # logprob(model, x, y) -> sum of log Pr(y | x) under that model: scalar per example
    # policy = the model we're training; ref_policy = frozen starting model (the KL anchor)

    # How much MORE likely each response is under the trained policy vs. the reference.
    # DPO's derivation shows the implicit reward is exactly beta * (logp_policy - logp_ref).
    chosen_logratio   = logprob(policy, prompt, y_chosen)   - logprob(ref_policy, prompt, y_chosen)
    rejected_logratio = logprob(policy, prompt, y_rejected) - logprob(ref_policy, prompt, y_rejected)

    # We want the chosen response's (implicit) reward to beat the rejected one's.
    # This is a logistic loss on the reward GAP -> a supervised objective, no sampling, no reward model.
    margin = beta * (chosen_logratio - rejected_logratio)
    loss = -F.logsigmoid(margin).mean()       # push margin > 0: chosen pulled up, rejected pushed down
    return loss

# Training loop is just supervised learning over a dataset of (prompt, chosen, rejected) triples:
#   for batch in preference_data:
#       loss = dpo_loss(policy, ref_policy, *batch); loss.backward(); opt.step()
# beta controls how far policy may drift from ref (the KL leash from RLHF, baked in).

Built on Prior Work

Results & Evidence

This is a textbook, so “evidence” is the cited empirical record it synthesizes, not new experiments. The robust, well-supported claims it relays:

• Scaling laws hold across orders of magnitude — loss is a smooth power law in N and D, validated repeatedly (Kaplan, Hoffmann/Chinchilla lineage). This is the strongest empirical regularity in the field.
• Self-supervised pre-training transfers — base models adapt to many downstream tasks with little task data, the foundation-model paradigm.
• DPO matches RLHF on many alignment benchmarks with far less infrastructure (from Rafailov et al.).
• Speculative decoding is provably lossless — the accept/reject rule keeps the output distribution identical to the big model’s, while delivering 2–3× speedups in practice.

What it does not establish (be honest with clients): the book is deliberately foundational, so it under-covers the bleeding edge (e.g., MoE training details, the latest long-context tricks, agentic tool-use, multimodal). Emergent abilities are described but their predictability remains contested in the literature (some are measurement artifacts of discontinuous metrics). And because it surveys rather than benchmarks, it doesn’t give you head-to-head numbers to pick between, say, GRPO vs. PPO vs. DPO for your specific task — it gives you the understanding to run that comparison yourself.

How You’d Use It

For someone running an AI services company building agentic and multi-agent systems, this book is the diagnostic toolkit for the question clients implicitly ask: “my LLM isn’t doing X — what do I change?” The five-stage model maps directly to interventions ranked by cost:

• Wrong tone / unsafe / not following instructions → alignment problem. Reach for SFT then DPO on a preference dataset. Cheapest reliable lever; you can offer “domain alignment” as a productized service (collect client preference pairs, run DPO, ship a steered model).
• Knows the wrong facts / outdated → context problem, not a training problem. Use RAG (context scaling) before you fine-tune. Far cheaper and more maintainable.
• Fails at multi-step reasoning → prompting + inference-time scaling. Add CoT, decomposition, self-consistency (sample N reasoning paths, majority vote), or a verifier that scores agent steps. This is directly applicable to making your MAS agents more reliable — a critic/verifier agent is exactly the “generate-and-verify thinking paths” pattern.
• Too slow / too expensive to serve → inference engineering. Speculative decoding, KV-cache reuse, grouped-query attention. These are buy-not-build (vLLM, TensorRT-LLM) but understanding them lets you spec hardware and quote latency honestly.

The most commercially relevant framing: the LLM-as-policy view unifies your agent work with alignment. Your multi-agent reward/critique loops are RLHF-shaped; understanding advantage, baselines, and KL penalties tells you why a critic agent stabilizes a generator agent and how to weight its feedback.

Build Your Own (Minimal Recipe)

You won’t pre-train a foundation model (that’s the buy side). The 80%-value build is a domain-aligned, reasoning-capable assistant on top of an open base model:

1. Start from an instruction-tuned open model (Llama / Qwen / Mistral). This gives you Chapters 1–2 for free.
2. SFT on your domain — a few thousand high-quality (instruction, ideal_response) pairs. Library: trl’s SFTTrainer + LoRA so it runs on one GPU.
3. DPO on preference pairs — collect (prompt, chosen, rejected) triples (chosen = good answer, rejected = a worse one). Use trl’s DPOTrainer. The hard part: dataset quality, not code. Garbage preferences → garbage alignment.
4. Add a reasoning layer at inference — CoT prompt + self-consistency (sample 5–10 paths, vote) for hard queries; route easy queries to a single pass to save cost.
5. Serve with speculative decoding via vLLM for latency.

The two genuinely hard parts: (a) building a clean preference dataset — this is the moat and the cost; (b) a good verifier/reward signal for reasoning tasks where correctness is checkable (math, code) is much easier than for subjective tasks. Start with checkable domains.

How to Improve It

Levers worth attacking, framed as testable experiments:

1. Swap DPO for GRPO on checkable tasks. DPO needs preference pairs; for math/code where you can verify answers, group-relative policy optimization (the DeepSeek-R1 recipe) reinforces correct samples directly. Test: same SFT base, DPO vs. GRPO on a math benchmark, compare pass@1.
2. Process- vs. outcome-reward verifiers. The book mentions verifiers; build a process reward model that scores each reasoning step, not just the final answer, and use it to prune a tree search. Hypothesis: better sample efficiency than outcome-only majority voting.
3. Adaptive inference-time compute. Don’t spend N samples on every query — train a small router to predict difficulty and allocate compute (1 pass for easy, tree search for hard). Directly cuts serving cost in a services business.
4. Multi-agent self-consistency. Replace single-model majority voting with a diverse ensemble of agents (different prompts/temperatures/models) and a debate/aggregator agent — the book notes ensembling gains come from diversity. Test against single-model self-consistency at matched compute.
5. KL-penalty scheduling in DPO. The beta leash is fixed; anneal it (loose early to explore, tight late to stabilize) and measure alignment-vs-capability tradeoff.

Glossary

• Self-supervised pre-training — learning from raw text by predicting hidden/next tokens; no human labels needed.
• Decoder-only Transformer — the GPT-style architecture that generates text left-to-right using causally-masked self-attention.
• QKV attention — mechanism where each token’s query is matched against all keys to produce weights over values; the core of the Transformer.
• Causal mask — sets attention to future tokens to −∞ so generation can only look backward.
• Multi-head attention — running attention in several sub-spaces in parallel so different heads specialize.
• Pre-norm / post-norm — where layer normalization sits relative to the residual add; pre-norm trains more stably at depth.
• Scaling law — empirical power-law relating model loss to parameters, data, or compute.
• Emergent ability — a capability absent in small models that appears past a size/compute threshold.
• In-context learning — the model learning a task from examples in the prompt, with no weight updates.
• Chain-of-thought (CoT) — prompting the model to emit intermediate reasoning steps before the answer.
• Self-refinement — generate → self-critique → revise, looped to improve an output.
• SFT (supervised fine-tuning) — fine-tuning on (instruction, response) pairs to make a base model follow instructions.
• RLHF — reinforcement learning from human feedback: train a reward model from human rankings, then optimize the policy against it.
• Policy (in RL) — for an LLM, the next-token distribution π(a|s) = Pr(y_t | x, y_<t).
• Reward model — a learned scorer that assigns a quality number to a model output, a proxy for human preference.
• Value function / baseline — expected future reward from a state; subtracted from reward to get advantage and reduce gradient variance.
• Advantage — reward minus baseline; how much better an action was than expected.
• PPO — proximal policy optimization; an RL algorithm that clips updates so the policy doesn’t move too far per step.
• KL penalty — a term keeping the trained policy close to the reference model, preventing reward-hacking.
• DPO (direct preference optimization) — aligns a model directly on preference pairs via a supervised-style loss, no reward model or RL loop.
• GRPO — group-relative policy optimization; reinforces good samples within a group, used for reasoning models like R1.
• Speculative decoding — a small draft model proposes tokens that a big model verifies in parallel; lossless speedup.
• KV-cache — stored keys/values from previous tokens so each new token doesn’t recompute attention over the whole prefix.
• Inference-time scaling — spending more compute at test time (longer thinking, search, ensembling) instead of training a bigger model.
• Self-consistency — sampling many CoT paths and taking the majority answer.
• RAG — retrieval-augmented generation; inject retrieved documents into the context to ground answers.