Understanding Deep Learning

TL;DR

Most ML resources are either cookbooks (here’s the PyTorch call) or research papers (here’s our one delta). This book sits in the gap: it explains the ideas underneath all of deep learning from one consistent notation and one mental model. The model is simple — every network is a function y = f[x, ϕ] with parameters ϕ; training is just picking ϕ to minimize a loss derived from maximum likelihood; and you minimize by computing gradients with backpropagation and stepping downhill with SGD/Adam. Once you internalize that loop, every “new” architecture (convolution, attention, residual connections, diffusion) becomes a different choice of how to wire the function — not a different paradigm. The headline payoff isn’t a benchmark number; it’s that you can read any 2024-era paper, sketch its architecture from memory, and judge whether its claims are real. The book is free as a PDF, used at dozens of universities, and is the single best “demystify the math” text for a builder who knows code but not the calculus underneath.

Problem & Motivation

If you run an AI services company, you live in a frustrating middle ground. You can wire up an agent, fine-tune a model, and ship a RAG pipeline — but when a client asks “why does our fine-tune overfit?” or “is diffusion or a VAE better for this?” or a vendor pitches some “novel architecture,” you’re reasoning by analogy and vibes, not from first principles. The two existing resource types both fail you:

• Cookbooks / framework docs teach you the API (nn.TransformerEncoder(...)) but treat the math as a black box. You can call attention; you can’t explain why the softmax is there or what breaks without it.
• Papers and graduate textbooks assume you already speak the notation — measure theory, variational inference, information geometry — and each uses different notation, so transformers and diffusion look like unrelated fields.

The concrete pain: deep learning looks like a zoo of unrelated tricks (why does a CNN share weights? why does a residual connection help? why does adding noise and removing it generate images?). Without a unifying frame, every new technique is something you memorize rather than derive. Prince’s thesis is that there is no zoo — there’s one animal wearing different costumes, and once you see the skeleton, the costumes are obvious. The book is also explicit that “understanding” is partly aspirational: nobody fully understands why over-parameterized networks generalize, and Chapter 20 is honest about exactly what remains unexplained.

What’s New (Core Contribution)

This is a textbook, not a research paper, so “novelty” means pedagogical and synthetic contributions, not a new algorithm:

• One notation for the entire field. Before: every architecture has its own symbols, so you can’t see that a transformer and a CNN are both f[x, ϕ] with structured weight matrices. Now: a single consistent notation (ϕ = parameters, Ω = weight matrices, β = biases, f[x, ϕ] = the function) spans Chapter 2 (linear regression) through Chapter 19 (RL). Self-attention, convolution, and a dense layer are visibly the same kind of object.
• Loss functions derived, not given. Before: “use cross-entropy for classification” is stated as a rule. Now: Chapter 5 derives every loss (MSE, cross-entropy, the diffusion objective, the VAE ELBO) from one recipe — pick a probability distribution over outputs, then maximize likelihood. MSE is maximum likelihood under a Gaussian; cross-entropy is maximum likelihood under a categorical. This is the single most clarifying idea in the book.
• Generative models unified. Before: GANs, VAEs, normalizing flows, and diffusion are taught as four separate fields. Now: Chapters 15–18 present them as four answers to one question (“how do you model Pr(x) well enough to sample from it?”), with explicit trade-offs (exact likelihood vs. sample quality vs. training stability).
• Honesty as a feature. Chapter 20 (“Why does deep learning work?”) catalogs what is not understood — double descent, why SGD finds good minima, why huge models generalize from little data. Most textbooks paper over this. Naming the open problems is itself a contribution for a practitioner deciding what to trust.

How It Works (Technically)

The whole book is one loop applied recursively. Here it is, then I’ll trace it through the hardest-earned example in the book (self-attention).

The universal recipe (Chapters 2–8):

1. Define a model y = f[x, ϕ] — a function from input x to prediction y, parameterized by ϕ. A shallow net is f[x] = ReLU[β + Ωx] then another linear layer. “Deep” just means you compose many such layers.
2. Define a loss L[ϕ] measuring how wrong the predictions are on training data. Chapter 5’s punchline: choose a distribution Pr(y|x), then L[ϕ] = -Σ log Pr(yᵢ | f[xᵢ, ϕ]). Plug in a Gaussian → you get mean-squared error. Plug in a categorical → you get cross-entropy. The loss is not arbitrary; it falls out of the distribution you assume.
3. Compute gradients ∂L/∂ϕ via backpropagation (Chapter 7) — the chain rule applied layer-by-layer, reusing intermediate results so it costs about the same as one forward pass.
4. Step downhill ϕ ← ϕ - α·∂L/∂ϕ with SGD, plus momentum and Adam (Chapter 6) to handle noisy, ill-conditioned gradients.
5. Generalize, don’t memorize (Chapters 8–9): measure on held-out data, add regularization, and confront the weirdness that bigger models often generalize better (double descent).

Everything after Chapter 9 is the same five steps with a cleverer choice of f in step 1.

Demystifying the key equations via self-attention (Chapter 12) — the deep trace. A transformer is “just” step 1 with a particular f. Take a sentence as N token vectors x₁…x_N, each of dimension D. Self-attention turns them into N new vectors of the same shape, where each output is a context-aware mix of all inputs. Three linear maps do the work:

• Values vₘ = β_v + Ω_v·xₘ. Plain English: project each token into a “what I contribute” vector. Operationally: one shared weight matrix Ω_v applied to every token (parameter sharing — same trick as a CNN).
• Queries and keys qₙ = β_q + Ω_q·xₙ and kₘ = β_k + Ω_k·xₘ. Plain English: each token emits a “what I’m looking for” (query) and a “what I offer” (key). These are the two halves of a soft database lookup — the names literally come from information retrieval.
• Attention weights a[xₘ, xₙ] = softmaxₘ(kₘᵀqₙ). Plain English: score how well token m’s key matches token n’s query (the dot product kₘᵀqₙ is a similarity), then softmax across all m so the scores are positive and sum to 1. This is the whole trick — the dot product measures relevance; the softmax turns relevance into a probability distribution over “where should token n look?”
• Output saₙ = Σₘ a[xₘ, xₙ]·vₘ. Plain English: each output token is a weighted average of all value vectors, weighted by relevance. Prince’s framing: attention is routing — it decides what fraction of each token’s value flows into each output.

Why this matters mechanically: the value/query/key step is linear and cheap, but the attention weights are themselves a nonlinear function of the input (because of the softmax over input-dependent dot products). So self-attention is a hypernetwork — one branch (queries·keys) computes the weights used by another branch (the value mixing). That’s the source of its power and its O(N²) cost: there’s one attention weight per ordered pair of tokens, so cost grows quadratically with sequence length but is independent of token dimension D. Every “long-context” trick in 2024 is an attack on that N².

In matrix form (how it’s actually implemented): stack tokens as columns of X, compute Q = β_q + Ω_q·X, K, V the same way, then Attention = softmax(KᵀQ) and output = V·Attention. Three matmuls and a softmax — that’s a transformer’s beating heart. A full transformer layer wraps this in multi-head attention (run several attention blocks in parallel on different learned projections), residual connections (Chapter 11), and layer normalization.

Architecture & data flow

flowchart TB
  subgraph RECIPE["The one recipe — every chapter is a variation"]
    M["1. Model: y = f[x, ϕ]"] --> L["2. Loss: -Σ log Pr(y|f[x,ϕ])<br/>(MSE / cross-entropy / ELBO / diffusion)"]
    L --> G["3. Gradients via backprop (chain rule)"]
    G --> S["4. SGD / Adam step: ϕ ← ϕ - α·∂L/∂ϕ"]
    S --> R["5. Regularize + measure on held-out data"]
    R -.repeat.-> M
  end
  RECIPE --> ARCH["Choose how to wire f[·]:"]
  ARCH --> CNN["Convolution → weight sharing over space (images)"]
  ARCH --> ATT["Self-attention → weight sharing + input-dependent routing (text)"]
  ARCH --> RES["Residual + BatchNorm → trainable depth"]
  ARCH --> GEN["Model Pr(x) → GAN / VAE / Flow / Diffusion (generation)"]
  ARCH --> RL["Reward instead of label → RL (policy gradient, actor-critic)"]

Interactive self-attention router. Each output token (column) is a weighted blend of all input value vectors; the heatmap shows the attention weights from softmax(KᵀQ). Hover a column to see how that output "routes" the inputs. This is schematic (random Q/K), but the shapes and the softmax-per-column behavior are exactly the real mechanism.

The algorithm, simplified

The book’s whole spine is this training loop. Everything else changes only model and loss.

# The universal deep-learning loop. Swap `model` and `loss` to get any chapter.
def train(model, data, loss_fn, lr=1e-3, steps=10_000):
    phi = model.init_params()                 # ϕ: all weights Ω and biases β
    opt = Adam(phi, lr)                        # momentum + per-param scaling (Ch. 6)
    for step in range(steps):
        x, y = data.sample_minibatch()        # SGD: estimate gradient on a subset
        y_hat = model.forward(x, phi)         # step 1: y = f[x, ϕ]
        L = loss_fn(y_hat, y)                 # step 2: -log Pr(y | f[x,ϕ])
        grads = backprop(L, phi)              # step 3: chain rule, one backward pass
        phi = opt.step(phi, grads)            # step 4: walk downhill
    return phi

# Self-attention as `model.forward` (Ch. 12) — the part that makes a transformer:
def self_attention(X, Wq, Wk, Wv):            # X: [D, N] tokens in columns
    Q, K, V = Wq @ X, Wk @ X, Wv @ X          # 3 linear maps (shared across tokens)
    scores = K.T @ Q                          # [N, N]: key·query similarity
    A = softmax(scores, axis=0)               # per-column: where should each token look?
    return V @ A                              # [D, N]: each output = weighted blend of values

Built on Prior Work

The book is a synthesis, so its “prior work” is the field itself. The value-add is putting these on one set of axes.

Results & Evidence

This is a textbook, so “evidence” is pedagogical efficacy and coverage, not a benchmark table — judge it accordingly:

• Coverage is current and complete. 21 chapters span the genuine 2023 frontier: transformers (encoder/decoder, BERT, GPT-3), diffusion models (full derivation), GNNs, and a modern RL chapter (policy gradient, actor-critic, offline RL). Few textbooks reach diffusion at all.
• Adoption is the real benchmark. It’s used in courses at many universities, the PDF is free, and it ships with Python notebooks and per-chapter problems — strong signal that the explanations actually land for learners.
• Honesty about limits is built in. Chapter 20 explicitly states what’s unexplained (double descent, generalization of over-parameterized nets, why SGD finds flat minima). This is rare and valuable.

What the evidence does NOT establish, and the caveats:

• Not a coding tutorial. The title is deliberate — “Understanding,” not “Implementing.” You’ll understand attention deeply but won’t learn production PyTorch idioms, distributed training, or serving here.
• 2023 snapshot. It predates the post-ChatGPT RLHF/RLAIF wave, mixture-of-experts at scale, modern long-context methods, and agentic patterns. The foundations are timeless; the frontier chapters are a 2023 photograph.
• Math is demystified, not removed. It needs first-year calculus, linear algebra, and probability (all reviewed in appendices). A reader allergic to any notation will still struggle — but the on-ramp is as gentle as this material gets.
• No empirical claims to scrutinize — there are no experiments of the author’s own to cherry-pick; the risk instead is omission (what got left out of a 540-page survey).

How You’d Use It

For someone running an AI services company, this book is a capability multiplier, not a line item. Concrete uses:

• Pre-sales credibility. When a prospect asks “should we fine-tune, RAG, or train from scratch?” you can reason from the loss-function and data-efficiency arguments (Ch. 8–9, 20) instead of guessing. That’s the difference between a vendor and a trusted advisor — and it closes deals.
• Architecture due diligence. A client or competitor pitches a “novel model.” With this book’s frame you can ask: what’s f? what loss? what’s the data assumption? Most “novel” pitches collapse to a known recipe with a tweak. You’ll spot snake oil fast (the reader-relevant superpower: honest hype detection).
• Debugging client models. “Our fine-tune overfits / won’t converge / generates garbage” maps directly to chapters: regularization (9), initialization and gradients (7), generative-model trade-offs (15–18). You debug from mechanism, not Stack Overflow roulette.
• Onboarding and team leveling. It’s the single best assignment for a new hire who can code but doesn’t know the math. Free PDF + notebooks + problems = a ready-made internal curriculum. Productize this: a “deep-learning literacy” track for client engineering teams is a sellable engagement.
• Grounding agentic work. Your multi-agent systems sit on top of models you treat as black boxes. The RL chapter (19) and the transformer chapters (11–12) are exactly the internals you need when an agent’s underlying model behaves unexpectedly, or when you’re evaluating fine-tuning vs. prompting for a specialist agent.

Where it slots in: this is the foundations layer under everything you already do. It won’t change your stack tomorrow; it changes the quality of every architectural decision you make for the next decade.

Build Your Own (Minimal Recipe)

You don’t “build” a textbook — but the highest-leverage move is to internalize its loop by reimplementing it. The smallest project that captures ~80% of the value:

Build a from-scratch training loop in NumPy, then add one transformer layer.

1. MLP + backprop in pure NumPy (Ch. 2–7). Implement forward, a hand-derived backprop, MSE loss, and plain SGD. Fit a sine curve. This forces you to feel the chain rule. ~150 lines.
2. Swap the loss to cross-entropy (Ch. 5) and classify two moons. Watch how the same loop handles a different task by changing only step 2. This is the book’s central lesson, learned by hand.
3. Add Adam and a held-out split (Ch. 6, 8). Plot train vs. test loss; deliberately overfit, then add L2 regularization and watch the gap close (Ch. 9).
4. Implement single-head self-attention (Ch. 12) — the self_attention function above is the whole thing. Verify each output column is a convex combination of value columns (weights sum to 1). This is the moment transformers stop being magic.

The two genuinely hard parts: (a) getting backprop’s index bookkeeping right by hand — use the book’s toy example as a unit test; (b) numerical stability in softmax and the log-likelihood (subtract the max before exp). Libraries to reach for: just NumPy for the learning version; then PyTorch (torch.autograd) once you trust your hand-derived gradients — diffing your NumPy gradients against autograd is the best confidence check there is.

How to Improve It

Treating the book as a system with limitations you could extend or pair with:

1. Bolt on a modern-LLM-training supplement. The gap is RLHF/DPO, instruction tuning, and MoE. Pair Ch. 19 (RL) with a focused read on policy-gradient-based preference optimization (PPO→GRPO/DPO) — that’s the bridge from “this textbook” to “how 2024 chat models are actually aligned.” A high-value internal doc for your team.
2. Add an inference/systems layer. The book is training-time. The commercial frontier is inference — KV-cache, quantization, speculative decoding, batching economics. These determine your unit costs. Build a companion module mapping each architecture chapter to its serving cost profile.
3. Make the notebooks agentic. The existing notebooks teach one model at a time. Wrap them in a small eval harness that swaps model/loss and auto-plots the comparison — turning the book’s “it’s all one loop” thesis into a runnable experiment grid. Reusable as a client demo.
4. Stress-test the “honesty” chapter against 2024 results. Chapter 20’s open problems (double descent, generalization) have newer partial answers (grokking, scaling-law mechanism work). A literature update keeps the most valuable chapter current.
5. Build the missing graph-RAG / retrieval connection. Ch. 13 (GNNs) and modern retrieval/graph-RAG are natural neighbors the book doesn’t connect. For an AI services shop doing knowledge work, linking message-passing on graphs to retrieval architectures is a differentiated offering.

Glossary

• f[x, ϕ] — the model as a function: input x, parameters ϕ (all the weights), output y. The whole book is choices of this function.
• ϕ (phi) — every learnable parameter in the network, lumped together; training = searching for the best ϕ.
• Ω / β — a weight matrix and a bias vector inside one layer (f = ReLU[β + Ωx]).
• Loss function — a number measuring total prediction error; derived from “negative log-likelihood” of the data under an assumed distribution.
• Maximum likelihood — pick parameters that make the observed data most probable; the source of MSE (Gaussian) and cross-entropy (categorical).
• Backpropagation — the chain rule run backward through the network to get ∂L/∂ϕ cheaply (one extra pass).
• SGD / Adam — gradient descent on minibatches (SGD) and an adaptive version with momentum and per-parameter scaling (Adam).
• Regularization — anything that discourages memorizing the training set (L2 penalty, dropout, early stopping) to improve generalization.
• Double descent — surprising phenomenon where test error drops, rises, then drops again as models get bigger; one of DL’s unexplained behaviors.
• Convolution — a layer that applies the same small weight set across all positions of an input (weight sharing → translation invariance); the CNN building block.
• Residual connection — adding a layer’s input to its output (x + f[x]) so gradients flow and deep nets train; the ResNet idea.
• Self-attention — turns each token into a weighted blend of all tokens, where weights come from query·key similarity via softmax; the transformer core.
• Query / Key / Value — three learned projections of each token: what it’s looking for, what it offers, and what it contributes. Borrowed from information retrieval.
• Softmax — squashes a vector of scores into positive numbers that sum to 1 (a probability distribution); makes attention “route.”
• Hypernetwork — a network whose one branch computes the weights used by another branch; self-attention’s weights are computed from the input this way.
• Multi-head attention — running several self-attention blocks in parallel on different learned projections, then concatenating — lets the model attend to several relationships at once.
• ELBO — Evidence Lower Bound; the loss VAEs maximize, a tractable proxy for the data likelihood.
• Diffusion model — a generator that learns to reverse a gradual noising process; sample noise, denoise step by step into an image.
• Policy gradient — RL method that treats the policy as a network and nudges its parameters toward higher-reward actions; the bridge from RL to the same gradient-descent loop.