Hierarchical Reasoning Model

TL;DR

Today’s reasoning LLMs “think out loud” by generating long chains of text tokens (chain-of-thought). That’s slow, data-hungry, and brittle — one wrong token can derail the whole answer. This paper proposes the Hierarchical Reasoning Model (HRM): two coupled recurrent neural networks, a slow high-level planner and a fast low-level worker, that iterate on an internal hidden state to “reason in latent space” — no words generated mid-thought. The trick that makes it work is hierarchical convergence (the fast module repeatedly settles to a sub-answer, then the slow module nudges it and restarts it) plus a one-step gradient that trains it with constant memory instead of the usual expensive backprop-through-time. The headline: with only 27M parameters and ~1000 training examples per task, no pretraining and no CoT, HRM gets 40.3% on ARC-AGI-1 (beating o3-mini-high’s 34.5% and Claude 3.7’s 21.2%), ~55% on extreme Sudoku, and ~74% on hard 30×30 mazes — tasks where the CoT baselines score 0%.

Problem & Motivation

The concrete pain: standard Transformers are computationally shallow. A fixed-depth Transformer can only do a bounded amount of sequential computation per forward pass — formally it sits in weak complexity classes (AC⁰/TC⁰) and is not Turing-complete. So it physically cannot run an algorithm that needs many dependent steps (deep search, backtracking, constraint propagation) in one shot.

The industry’s workaround is Chain-of-Thought: make the model write its intermediate steps as text, feeding each step back in, so “depth” comes from generating more tokens. The authors call CoT “a crutch, not a solution”:

• Brittle. Reasoning is tied to a human-defined sequence of language steps; one misstep or misordering derails the whole thing.
• Data-hungry. You need lots of worked examples / RL to teach the step pattern.
• Slow. Hard problems mean thousands of generated tokens, i.e. high latency.

The deeper claim: language is for communication, not the substrate of thought. Brains reason in a latent space without constantly translating to words. So the goal is latent reasoning — do the computation inside the hidden state. But naive latent reasoning hits two walls: stacking many layers causes vanishing gradients, and recurrent networks (the natural fix) suffer premature convergence (the hidden state stops changing, so extra steps are wasted) and require backpropagation-through-time (BPTT), which stores every intermediate state → O(T) memory, small batches, poor GPU use, and is biologically implausible. HRM is the attempt to get deep, stable, cheap-to-train latent reasoning.

What’s New (Core Contribution)

1. Two-timescale coupled recurrence (the architecture). Before: a single RNN that converges and stalls, or a fixed-depth Transformer. Now: a high-level module (H) updates slowly and sets strategy, a low-level module (L) updates fast and does the grind. This gives effective depth = N×T steps from a small model.
2. Hierarchical convergence (the dynamics). Before: RNNs converge once then go inert. Now: L converges to a local equilibrium within each cycle, then H updates and resets L’s context, kicking off a fresh convergence toward a new equilibrium — a sequence of nested, stable computations instead of one dying one.
3. One-step gradient (the training trick). Before: BPTT, O(T) memory, unroll the whole sequence. Now: differentiate only at the fixed point using a one-term approximation of the implicit-function-theorem gradient (Deep Equilibrium Models lineage) → O(1) memory, no unrolling, ~5 lines of PyTorch.
4. Adaptive Computation Time via Q-learning (the “think longer when needed”). A learned halting head decides, per problem, how many forward passes (“segments”) to spend — fast for easy, slow for hard — and lets you scale test-time compute by just raising a limit, no retraining.

Repackaged-vs-novel: deep equilibrium models, ACT, and multi-timescale RNNs (Clockwork RNN) all pre-exist. The genuinely new combination is hierarchical convergence solving premature convergence so the one-step gradient actually trains a usefully deep recurrent reasoner from tiny data.

How It Works (Technically)

Think of HRM as a CPU with two clocks. The L-module is the fast clock (an inner loop doing detailed work); the H-module is the slow clock (an outer loop that reads L’s result, updates the plan, and restarts the inner loop). Both are encoder-only Transformer blocks (Llama-style: RoPE, GLU, RMSNorm, no biases). They are combined by simple element-wise addition of their inputs.

The four learnable parts: input net f_I, low-level net f_L, high-level net f_H, output head f_O.

One forward pass = N cycles × T low-level steps each. Concretely with N=2, T=2 you get 4 internal steps. Trace one input:

1. Embed. x̃ = f_I(x) — turn the input grid (e.g. a flattened 9×9 Sudoku, 81 tokens) into a working representation.
2. Inner loop (L runs T times, H frozen). At each step i: z_L^i = f_L(z_L^{i-1}, z_H^{i-1}, x̃) In plain English: the worker updates its scratchpad given its own last scratchpad, the current plan (held fixed), and the problem. Over T steps z_L settles toward a local equilibrium — a sub-answer consistent with the current plan.
3. Outer step (H updates once per cycle). z_H^i = f_H(z_H^{i-1}, z_L^{i-1}) only when i ≡ 0 (mod T), else z_H is unchanged. In plain English: once the worker has converged, the planner reads the result and revises strategy. This new z_H is a fresh context that restarts L’s convergence next cycle toward a different equilibrium. This restart-instead-of-stall is hierarchical convergence — it’s why activity (the “forward residual”) stays high across many steps instead of decaying to zero like a vanilla RNN.
4. Read out. After N cycles, ŷ = f_O(z_H^{NT}) decodes the final high-level state into output tokens (the solved grid).

Why this gives depth cheaply: a normal RNN’s useful computation ends after ~T steps (convergence). By resetting L every cycle, HRM chains N such converged computations → effective depth N×T, but the model is still tiny (27M params, 8 effective layers).

The training trick — one-step gradient (demystifying the math). If L truly reaches a fixed point z_L* = f_L(z_L*, z_H, x̃), you don’t have to backprop through all T steps. The Implicit Function Theorem gives the exact gradient of a fixed point as (I − J_F)^{-1} ∂F/∂θ, where J_F is the Jacobian of the update map. Inverting (I − J_F) is expensive, so they use the Neumann series (I − J_F)^{-1} = I + J_F + J_F² + … and keep only the first term (≈ I). The result: the gradient is just the derivative of the last update of each module, treating everything before it as a constant. Operationally that means: run the whole N×T loop under torch.no_grad(), then do one more L-step and one H-step with gradients, and backprop only through those. Gradient path: output head → final H state → final L state → input embedding. Cost: O(1) memory. This is the whole reason you can train a “deep” recurrent reasoner on a normal GPU.

Deep supervision. One forward pass is a “segment.” They run several segments per example, applying the loss after each segment and — crucially — detaching the hidden state between segments (z = z.detach()). So gradients never flow across segments; each segment is supervised independently. This gives frequent feedback to H and acts as regularization (cheaper and more stable than Jacobian-based DEQ regularizers).

Adaptive Computation Time (ACT) — the RL part you should understand. How many segments to run? A learned Q-head reads the final H-state and outputs two values: Q̂_halt and Q̂_continue (via a sigmoid). This is framed as a tiny Markov Decision Process: state = current hidden state, actions = {halt, continue}. Reward for halting = 1 if the prediction is correct else 0; reward for continuing = 0. The Q-learning targets are Ĝ_halt = 1{ŷ = y} and Ĝ_continue = max(Q̂_halt, Q̂_continue) of the next segment (standard bootstrapped Q-learning). The model halts when Q̂_halt > Q̂_continue (past a random minimum) or hits a max. The loss adds a binary-cross-entropy term on the Q-head to the sequence loss: L_ACT = LOSS(ŷ, y) + BCE(Q̂, Ĝ). Why it’s stable without replay buffers / target networks (which deep Q-learning usually needs): the architecture is Post-Norm with RMSNorm and trained with AdamW, which bounds the weights — recent theory (Gallici et al.) says bounded weights + weight decay + post-norm is enough for Q-learning to converge. Practical payoff: at inference you can just raise M_max and the model “thinks longer” → more accuracy on Sudoku, for free, no retraining.

Architecture & data flow

flowchart TD
  X[Input grid x] --> EMB[Input net f_I → x̃]
  EMB --> L
  subgraph CYCLE[One high-level cycle, repeated N times]
    direction TB
    L[L-module: T fast steps<br/>z_L = f_L of z_L, z_H, x̃<br/>converges to sub-answer] --> H[H-module: 1 slow step<br/>z_H = f_H of z_H, z_L<br/>revises plan, resets L]
    H -.fresh context restarts L.-> L
  end
  CYCLE --> OUT[Output head f_O → ŷ]
  OUT --> Q{Q-head: halt or continue?}
  Q -->|continue| SEG[detach state → next segment]
  SEG --> EMB
  Q -->|halt| FINAL[Final prediction]

Schematic of hierarchical convergence: the fast L-module (blue) repeatedly settles to a local equilibrium within each cycle; when the slow H-module (orange) updates, it resets L's context and a new convergence phase begins — so the "forward residual" (activity) spikes back up instead of decaying to zero like a plain RNN. Illustrative, not the paper's exact numbers.

The algorithm, simplified

# HRM forward pass + deep-supervision training loop (the whole idea in ~25 lines)
def hrm(state, x, N=2, T=2):
    x = input_embedding(x)            # f_I: tokens -> working representation
    zH, zL = state                    # two recurrent hidden states
    with torch.no_grad():             # run almost everything WITHOUT gradients
        for i in range(N * T - 1):
            zL = L_net(zL, zH, x)     # fast worker: converge toward a sub-answer
            if (i + 1) % T == 0:      # once per cycle...
                zH = H_net(zH, zL)    # slow planner updates, resetting L's context
    # ---- 1-step gradient: only the LAST update of each module is differentiated ----
    zL = L_net(zL, zH, x)             # final L step (with grad)
    zH = H_net(zH, zL)                # final H step (with grad)
    return (zH, zL), output_head(zH)  # O(1) memory: no BPTT, no unrolling

for x, y_true in train_dataloader:
    z = z_init
    for seg in range(N_supervision):          # deep supervision: several segments
        z, y_hat = hrm(z, x)
        loss = softmax_cross_entropy(y_hat, y_true)  # (+ Q-head BCE for ACT)
        z = z.detach()                # KEY: cut the graph between segments
        loss.backward(); opt.step(); opt.zero_grad()

Built on Prior Work

Results & Evidence

Headline (Figure 1), all from scratch, ~1000 examples, no pretrain, no CoT:

• ARC-AGI-1: 40.3% vs o3-mini-high 34.5%, Claude 3.7 (8K) 21.2%, DeepSeek R1 21.0%, Direct-pred baseline 15.8%.
• ARC-AGI-2: ~5% (small but the giant CoT models are also near-floor here, e.g. ~3% and below).
• Sudoku-Extreme (9×9): ~55% — CoT models score 0%.
• Maze-Hard (30×30 optimal path): ~74.5% — CoT models score 0%.

Supporting evidence worth trusting:

• Depth, not width, is what matters (Fig 2): widening a Transformer does nothing on Sudoku; deepening helps but saturates — HRM keeps gaining with depth. This directly motivates the architecture.
• ACT works (Fig 5): matches fixed-compute accuracy while using far fewer average steps, and a model trained at M_max=8 keeps improving when run at M_max=16 (genuine test-time scaling).
• Brain correspondence (Fig 8): after training, the H-module’s effective dimensionality (Participation Ratio ≈ 90) is ~3× the L-module’s (≈ 30), echoing the mouse-cortex hierarchy (~2.25×); an untrained net shows no separation, so it’s learned, not architectural.

Caveats — be honest:

• Narrow, puzzle-only. Sudoku, mazes, ARC grids. No language, no open-ended tasks, no transfer. This is a specialist solver, not a general model.
• One task per model. Each result is a network trained on that one task’s ~1000 examples (with heavy augmentation — e.g. ARC uses 1000 augmented variants per test input, then votes). It’s not a single model doing all three.
• ARC-AGI-2 is ~5% — the “AGI” framing is aspirational; on the harder benchmark it’s near the floor.
• Brain-correspondence is correlational, as the authors explicitly state; no causal intervention.
• Interpretability is preliminary — the “it does DFS / hill-climbing” claims come from eyeballing intermediate decodes (Fig 7), not mechanistic proof.

How You’d Use It

For an AI-services shop, HRM is not a chatbot replacement — it’s a blueprint for a small, fast, on-prem “reasoning solver” for structured, well-defined problems where you have input→output examples but reasoning is deep:

• Combinatorial / constraint problems for clients: scheduling, routing, layout, configuration, puzzle-like optimization, board/grid games, circuit or floor-plan checks. If you can frame it as “grid/sequence in → grid/sequence out” with a verifier for correctness, HRM-style training applies.
• Edge / cost-sensitive deployments: 27M params runs cheaply and offline. A client who can’t or won’t send data to a frontier API, and needs a deterministic-ish solver, is the sweet spot. This is a real moat vs “we call the OpenAI API.”
• Latency-critical reasoning: single forward pass (plus a few ACT segments) beats generating thousands of CoT tokens.
• As a tool inside a multi-agent system: your orchestrator (the LLM agents you already build, ARC MAS-style) handles language, planning, and tool selection; an HRM-style solver becomes a callable specialist tool for the hard combinatorial sub-step the LLM is bad at. That’s the most realistic integration: HRM as a tool, not as the brain.

Reality check on selling it: training a bespoke HRM per problem class is an R&D engagement, not a weekend. The payoff is a defensible, low-cost capability for a specific vertical.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value (the reference repo is github.com/sapientinc/HRM):

1. Pick a task with a verifier. Sudoku is the canonical starter: easy to generate, exact correctness check, deep reasoning. Represent input and solution as flattened token grids.
2. Build two identical Transformer-encoder stacks (H and L), small (e.g. 4 layers each, ~256–512 hidden). Use RMSNorm + RoPE + GLU if you can; element-wise add the inputs to each module.
3. Implement the N×T loop exactly as the pseudocode: everything under no_grad, then one gradient-carrying L-step and H-step. This is the load-bearing trick — get it right and training is cheap.
4. Add deep supervision: loop several segments, detach() the state between them, apply the loss each segment.
5. (Optional, do last) Add the ACT Q-head: start with a fixed number of segments to validate the core works; bolt on Q-learning halting only once you want adaptive compute.
6. Augment hard. Their data efficiency leans on heavy augmentation (permutations, rotations, flips) + voting at test time. Budget for this.

The two genuinely hard parts: (a) getting the one-step gradient + hierarchical convergence stable — if L doesn’t actually converge within T steps, the IFT approximation is wrong and training is shaky; (b) the Q-learning halting — even with the Post-Norm/AdamW stabilizer, RL halting is fiddly, which is why you add it last. Libraries: PyTorch is all you need; no RL framework required for the tiny Q-head.

How to Improve It

1. Replace element-wise addition with gating/cross-attention between H and L. The authors flag this as future work; a learned gate (or H→L cross-attention) likely improves how the plan steers the worker.
2. Multi-task / shared-backbone HRM. Train one model across Sudoku+maze+ARC with task-conditioning tokens, and measure transfer. Current results are one-model-per-task; generality is the obvious frontier.
3. Pair HRM with an LLM front-end so it accepts natural-language problem statements (LLM parses → grid encoding → HRM solves → LLM verbalizes). This turns a puzzle solver into a client-facing reasoning tool.
4. Add hierarchical memory / linear-attention state (the authors note full attention is used “for simplicity”). For long-horizon tasks, a multi-timescale memory could extend effective horizon and cut cost.
5. Test the causal role of dimensionality. Constrain the H-module’s dimensionality during training and see if reasoning degrades — turns the correlational brain-correspondence claim into a testable mechanism, and could yield a regularizer that forces the useful hierarchy.
6. Curriculum on difficulty (backtrack count). Since they measure puzzle hardness by solver backtracks, train easy→hard to improve sample efficiency and the ceiling on extreme instances.

Glossary

• Chain-of-Thought (CoT) — prompting an LLM to emit intermediate reasoning steps as text; “depth” via more tokens.
• Latent reasoning — doing the reasoning inside the hidden state vectors instead of in generated text.
• Recurrent module (RNN-style) — a network that feeds its own output back as input over steps, building up computation over time.
• Effective computational depth — how many dependent compute steps a model can perform per forward pass; HRM gets N×T.
• Hierarchical convergence — HRM’s core dynamic: the fast module settles, the slow module updates and resets it, repeating to avoid the stall a plain RNN hits.
• Premature/early convergence — an RNN’s hidden state stops changing, wasting later steps; HRM’s main enemy.
• BPTT (Backprop Through Time) — standard RNN training that unrolls every step; needs O(T) memory.
• Deep Equilibrium Model (DEQ) — a model defined by a fixed point; you differentiate at the fixed point instead of unrolling.
• Implicit Function Theorem (IFT) — the calculus result that lets you compute the gradient of a fixed point without unrolling.
• Neumann series — (I−J)^{-1} = I + J + J² + …; HRM keeps only the first term → the “1-step gradient.”
• Jacobian (J_F) — matrix of partial derivatives of the update map; appears in the exact fixed-point gradient.
• Deep supervision — apply the loss after each segment and detach state between segments, so gradients don’t cross segments.
• Adaptive Computation Time (ACT) — let the model choose how many compute steps/segments to spend per input.
• Q-learning — RL method that learns the value Q(state, action) of each action; here, halt vs continue.
• Markov Decision Process (MDP) — formalism of states, actions, rewards, transitions used to set up the Q-learning.
• Participation Ratio (PR) — a measure of how many dimensions a representation effectively uses; higher = richer.
• Post-Norm / RMSNorm — normalization placement/variant that, with AdamW, keeps weights bounded and stabilizes the Q-learning.
• ARC-AGI — abstraction-and-reasoning benchmark of few-shot grid puzzles; a proxy for fluid intelligence.
• Turing-complete — capable, given enough time/memory, of simulating any computation; standard Transformers are not, HRM aims to be.