Recursive Language Models

TL;DR

LLMs choke on long inputs: they have hard context-window ceilings, and even below those ceilings their accuracy rots as prompts get longer (“context rot”). The usual fix — summarize/compact the context when it gets too big — throws away detail that dense tasks actually need. This paper flips the framing: don’t feed the long prompt into the network at all. Put it in a Python REPL as a string variable, give the model a sub_rlm() function that calls a fresh LLM on any slice, and let the model program its way through — peeking, regex-filtering, chunking, looping, and stitching results back together. The result, called a Recursive Language Model (RLM), processes inputs two orders of magnitude past the base model’s window, beats vanilla GPT-5 and standard long-context scaffolds (retrieval, compaction) by double-digit margins on four diverse tasks, and costs about the same. They also show you can train a tiny model (Qwen3-8B) to be natively recursive with just 1,000 examples, lifting it 28.3% on average.

Problem & Motivation

Here is the concrete pain. You hand GPT-5 a 1-million-token codebase, or a 10M-token document corpus, and ask a question whose answer depends on details scattered everywhere. Two things break:

1. The hard wall. GPT-5’s context window is 272K tokens. Past that, the input literally does not fit. You’re stuck.
2. The soft wall (context rot). Even within the window, accuracy degrades steeply as the prompt grows — and it degrades faster for harder tasks. The paper’s key reframe: the “effective context window” isn’t a property of the model alone, it’s a property of model × task. A needle-in-a-haystack lookup (find one phrase) stays easy at 1M tokens. A task that must touch every line (OOLONG) degrades far earlier. A task that must reason over every pair of lines (their new OOLONG-Pairs) collapses to near-zero F1 even at modest lengths — GPT-5 scores 0.1%.

The standard production answer is context compaction: when the running context exceeds a threshold, summarize it and keep going. This is what most agent frameworks (and OpenAI’s own tooling) do. The fatal assumption baked in: details that appeared early can safely be forgotten to make room. That’s fine for chatty agent trajectories; it’s poison for tasks needing dense access across the whole input. Retrieval (BM25, embeddings) helps when the answer lives in a few findable chunks, but falls apart when the task needs aggregation over the whole thing.

So the field has been trying to push the wall outward — longer windows via architecture and training. This paper asks a different question: can we scale effective context by orders of magnitude without touching the base model at all?

What’s New (Core Contribution)

1. The prompt-as-environment reframe. Before: the long prompt is input to the neural net — it must fit in the window and it pollutes attention. Now: the long prompt is a variable in an external REPL the model interacts with symbolically. The model sees only constant-size metadata (length, a prefix, how to access it) and writes code to examine the rest. This is the whole trick, and it’s almost embarrassingly simple.

2. Symbolic recursion via sub_rlm(). Before: “sub-agent” / self-delegation approaches let an LLM call itself, but the parent has to verbalize each sub-call autoregressively in its own output — so you can only launch a handful, hand-placed. Now: the model writes a Python loop that calls a fresh LLM on O(|P|) or even O(|P|²) programmatically-constructed slices, storing results in variables. The number of sub-calls is bounded by code, not by the parent’s token budget. This is the difference between “delegate 3 tasks I wrote out” and “run this LLM over all 1,000 chunks in a for-loop.”

3. Unbounded output, too. Because the final answer is built up in a REPL variable (concatenating sub-call outputs), the output length is no longer capped by the base model’s max generation length. The model returns a variable, not a single autoregressive stream.

4. The first natively-recursive model + a cheap training recipe. They fine-tune Qwen3-8B on 1,000 trajectories (distilled from a bigger model acting as an RLM, on an unrelated task domain) and get a +28.3% median lift across four eval tasks, with lower cost from better decisions. Evidence that “being an RLM” is a general, trainable skill — a potential new axis of scale.

The honest read: contributions 1–3 are a clean, well-argued systems idea rather than new math. The cleverness is in identifying exactly the three design choices (handle to the prompt, programmatic recursion, variable-based output) that separate this from the deceptively-similar agent scaffolds everyone already builds. Contribution 4 is a small but suggestive proof-of-concept, not a trained frontier model.

How It Works (Technically)

An RLM exposes the same interface as a plain LLM: string in, string out. Internally, given a prompt P:

1. Initialize a REPL. Spin up a persistent Python Read-Eval-Print Loop. Set P as a string variable inside it. Also inject a function sub_rlm(new_prompt) -> str that runs a fresh LLM (or another RLM) on whatever string you pass it.

2. Seed the root model with metadata only. The root model M is not shown P. It’s shown constant-size metadata: len(P), a short prefix, and instructions on how to access P from code. This is the load-bearing constraint — it forces the model to use variables and code instead of dumping the prompt into its own window.

3. Loop. Each iteration: the root model emits a block of Python code. The REPL executes it against the persistent state, returning whatever was printed to stdout. Only constant-size metadata about that stdout (a prefix + length) goes back into the model’s history — not the full output. So the model’s own context stays small no matter how big the data it’s manipulating.

4. Terminate. When the model sets a special Final variable in the REPL, the loop stops and that variable’s value is returned as Y.

That’s Algorithm 1. The paper’s clearest teaching move is Algorithm 2 — a “deceptively similar” scaffold that looks the same but is far weaker, isolating three flaws:

• Flaw 1 — prompt lives in the context window. Algorithm 2 puts P directly into the model’s history (hist). It instantly inherits M’s window limit and is forced to fall back on compaction. RLM never copies P into the window; it only ever holds a handle.
• Flaw 2 — output is autoregressive. Algorithm 2 finishes by having the model generate the answer directly (a Finish action), so output length is capped by the window. RLM returns a variable, so output is unbounded.
• Flaw 3 — no programmatic recursion. Algorithm 2 has a “call sub-LLM” action and a “run code” action, but they’re separate — it can’t invoke the sub-LLM from inside a loop in code. It can delegate a few verbalized tasks; it cannot launch O(|P|²) sub-calls programmatically. This is the most important difference.

Why does this beat brute force on cost? Intuition: the model filters context without seeing it. It uses regex to grab only chunks containing “festival” or a name it has a prior about (“La Union”), processing far fewer tokens than the full input. The base model’s world-knowledge priors prune the search space. So you pay for the tokens you actually need, not all 10M.

Recursion depth. They use depth one — sub-calls are plain LLM calls, not RLMs-of-RLMs. The framework allows deeper recursion (sub_rlm calling sub_rlm), but they didn’t need it for these benchmarks. There’s also a natural iteration bound: if each root turn is trimmed to c tokens and the window is K, you get at most K/c root iterations — each of which can fan out to arbitrarily many sub-calls.

Architecture & data flow

flowchart TB
  U[User: arbitrary-length prompt P] --> R[REPL environment E]
  R -->|P stored as variable| V[(P: string in memory)]
  R -->|inject helper| F["sub_rlm(prompt) -> str"]
  R -->|metadata only: len, prefix| M[Root LLM M]
  M -->|emits Python code| EX[Execute in REPL]
  EX -->|regex / slice / chunk| V
  EX -->|loop over slices| F
  F -->|fresh LLM per slice| SUB[Sub-LLM calls]
  SUB -->|results stored in vars| V
  EX -->|stdout prefix+length only| M
  M -->|sets Final variable| OUT[Return Final as response Y]

Schematic of "context rot": move the length slider and switch task complexity. A plain LLM (orange) holds up on constant-complexity lookups but collapses on linear/quadratic tasks as length grows; the RLM (blue) stays flat because it never ingests the whole prompt. Curves are illustrative of the paper's Figure 1 pattern, not exact data.

The algorithm, simplified

# The RLM loop. M is the base model; sub_rlm() runs a fresh LLM on any slice.
def rlm(P, M, max_iters=12, trim=2000):
    repl = REPL()                       # persistent Python environment
    repl.set("P", P)                    # the long prompt is JUST a variable
    repl.define("sub_rlm", lambda s: M(s))   # depth-1: sub-call = plain LLM call

    # The root model NEVER sees P — only constant-size metadata about it.
    hist = [f"P is a string, len={len(P)} chars. Prefix: {P[:200]!r}. "
            f"Use P and sub_rlm() in code. Set `Final` when done."]

    for _ in range(max_iters):
        code = M(hist)                   # root model writes Python (the 'reasoning')
        stdout, state = repl.run(code)   # execute against persistent state

        # KEY: feed back only a constant-size summary of stdout, not the whole thing.
        hist.append(code)
        hist.append(f"stdout(len={len(stdout)}): {stdout[:trim]}")

        if "Final" in state:             # model signals completion via a variable
            return state["Final"]        # output can be arbitrarily long
    return repl.get("Final")

# Example of code the ROOT model might emit on its own (this is generated, not written by us):
#   chunks = [c for c in P.split("\n") if re.search(r"festival|La Union", c)]
#   notes  = [sub_rlm(f"Does this line answer the query? {c}") for c in chunks]  # O(|P|) fan-out
#   Final  = synthesize(notes)

The contribution is almost entirely in two lines: repl.set("P", P) (prompt-as-variable) and the sub_rlm loop the model writes (programmatic recursion). Everything else is plumbing.

Built on Prior Work

The lineage is honest: every ingredient (REPL agents, sub-calls, distillation) existed. The novelty is the precise combination and the framing that the prompt itself belongs in the environment.

Results & Evidence

Setup. Four tasks chosen to vary in how processing cost scales with length: S-NIAH (constant — find one needle), BrowseComp-Plus (multi-hop QA over 1K docs, 6–11M tokens), OOLONG (linear — touch every line), and OOLONG-Pairs (quadratic — reason over every pair; their new benchmark). Base models: GPT-5 (272K window) and Qwen3-Coder-480B. Baselines: base model, CodeAct+BM25, CodeAct+sub-calls, summary agent.

Headline numbers:

• Past the wall: On BrowseComp-Plus (6–11M tokens, far beyond any window), base GPT-5 and CodeAct score 0% (input doesn’t fit); RLM(GPT-5) hits 91.3% at ~$0.99/query — beating retrieval and summarization baselines by >29%.
• Below the wall, still better: On OOLONG (fits in window), RLM beats base GPT-5 by +28.4% and Qwen3-Coder by +33.3%.
• The dramatic one: On OOLONG-Pairs, base GPT-5 and Qwen3-Coder score <0.1% F1 — total failure. RLM lifts them to 58.0% and 23.1%. This is the emergent-capability result.
• Cost: Median RLM run is cheaper than the median base run for GPT-5, and up to 3× cheaper than the summary agent — because it views context selectively. But variance is high: tail runs can be much more expensive than any single base call (long trajectories).
• Ablations: Remove sub-calls but keep the REPL → still scales past the window (REPL is what beats the length wall), but loses 10–59% on information-dense tasks (recursion is what handles density). Both pieces matter, for different reasons.
• Training: RLM-Qwen3-8B (1,000 distilled samples from an unrelated domain) beats base Qwen3-8B-as-RLM by +28.3% average and approaches vanilla GPT-5 on three tasks, at lower cost.

What the evidence does NOT establish:

• Narrow task surface. Four tasks, two of them synthetic (S-NIAH, OOLONG-Pairs). The authors admit “natural long-context tasks remain under-explored.” No coding-agent-in-the-wild, no real legal/financial document workloads.
• Cost tail is real risk. “Comparable cost at the median” hides outliers that blow up. For a budget-sensitive client offering, the p95 matters more than the median.
• Depth-1 only. They never actually exercise deep recursion; the O(|P|²) claim is a capability of the framework, demonstrated mostly via OOLONG-Pairs.
• Sequential, blocking calls. Their implementation runs sub-calls serially, so latency numbers are pessimistic but also un-optimized — runtime claims are caveated by the authors themselves.
• Model-dependent behavior. GPT-5 is conservative with sub-calls; Qwen3-Coder fans out aggressively (and needed a prompt line warning it not to). The scaffold’s behavior is not uniform across models.

How You’d Use It

This maps almost directly onto an AI-services practice — it’s a scaffold, not a model, so it works with whatever frontier API you’re already billing through.

• Document-heavy QA / “chat with our entire corpus.” Clients constantly want Q&A over a folder of contracts, a full codebase, years of tickets — corpora that blow past any window. Today you reach for RAG and apologize when aggregation questions fail (“how many of our 400 contracts have an auto-renewal clause?”). An RLM answers those by looping a sub-call over every chunk. This is the single most sellable use case.
• Replacement for compaction in long-running agents. If you run agents that summarize-when-full, you’re silently losing early detail. Swapping the “memory” for a prompt-as-variable REPL preserves it. Good fit for multi-hour research or audit agents.
• Long output generation. Reports, migrations, per-item transformations over a big list — anything where the answer is longer than the model’s max generation. The variable-stitching trick removes that ceiling.
• A premium tier. “Standard RAG” vs. “deep analysis (RLM)” is a natural pricing split: RLM costs more variance but answers questions RAG simply can’t. The catch you must underwrite: cost variance. Bound iterations and sub-call counts, and meter aggressively, or one pathological query eats the margin on ten good ones.

Where it does not belong: short prompts. The paper finds RLM is slightly worse than the base model on small inputs (extra overhead, no benefit). Route by input length.

Build Your Own (Minimal Recipe)

You can stand up a working RLM in an afternoon. ~80% of the value comes from the REPL + prompt-as-variable; sub-calls add the dense-task power.

Components:

1. A sandboxed Python REPL with persistent state. Reach for a code-execution sandbox (E2B, Modal, or a locked-down subprocess). Sandboxing is non-negotiable — the model writes and runs arbitrary code.
2. A loader that puts the user prompt into the REPL as a string variable and exposes sub_rlm(s).
3. The root loop: call the model with metadata-only history → get code → execute → feed back only a trimmed stdout summary → repeat until Final is set.
4. A system prompt teaching the model: it has a variable P, here’s how to peek/slice it, here’s sub_rlm, set Final to finish, don’t try to print all of P.

Build order: (1) REPL + variable injection, (2) the metadata-only history discipline — this is the part people get wrong; if you ever paste P back into history, you’ve rebuilt the broken Algorithm 2, (3) the system prompt, (4) sub_rlm, (5) iteration + sub-call caps for cost control.

The two genuinely hard parts:

• The metadata discipline. Every temptation is to show the model more. You must show it less — only lengths and prefixes — or you lose the entire benefit.
• Cost/iteration governance. Without hard caps on root iterations and total sub-calls, a confused trajectory can fan out into thousands of API calls. Budget guards are core, not optional.

Models: Use a strong root (GPT-5, Claude, a frontier reasoner) and a cheaper sub-call model (the paper uses GPT-5-mini for sub-calls, GPT-5 as root) — sub-calls are short-horizon and dominate volume, so cheap-sub/expensive-root is the cost sweet spot.

How to Improve It

1. Parallel / async sub-calls. The paper runs sub-calls serially and flags this. Fan out sub_rlm over chunks with asyncio or a worker pool — the O(|P|) loop is embarrassingly parallel. Likely the biggest single latency win, testable immediately.
2. Smarter decomposition than uniform chunking. They observed only naive strategies (split-by-newline, regex). Give the model a semantic chunker, or let it build an index variable first — could cut sub-calls (cost) while improving aggregation quality. Measurable on OOLONG-Pairs.
3. Cost-aware policy / budgeting. Add a per-trajectory token budget the model can see and reason about, so it trades thoroughness against cost. Attacks the p95-cost weakness head-on — the thing that scares a services business.
4. Deeper recursion. They cap at depth 1. Let sub_rlm itself be an RLM for genuinely hierarchical inputs (a corpus of long documents, each itself huge). Test on nested-structure tasks. Risk: cost compounds, so pair with #3.
5. RL instead of distillation for native RLMs. The training recipe is supervised distillation of trajectories. Their own framing — “RLM trajectories are a form of reasoning” — invites GRPO/PPO-style RL with a task-reward, which could discover decomposition strategies better than the teacher’s. Bigger lift, bigger effort.
6. Hybrid retrieve-then-recurse. Use cheap retrieval to pre-filter to a candidate set, then RLM-recurse only over those. Best of both: retrieval’s cheap pruning + RLM’s aggregation. Directly targets the cost tail.

Glossary

• RLM (Recursive Language Model) — an inference scaffold that puts the prompt in a REPL as a variable and lets the model write code that recursively calls an LLM on slices of it.
• REPL — Read-Eval-Print Loop; a live programming environment with persistent state where each code block runs against the variables left by previous blocks.
• Context window (K) — the max number of tokens a model can attend to at once; GPT-5’s is 272K.
• Context rot — the empirical degradation of model accuracy as the prompt gets longer, even when it still fits in the window.
• Effective context window — the usable length for a given task; the paper argues it depends on task complexity, not just the model.
• Context compaction / condensation — summarizing/truncating accumulated context once it exceeds a threshold; lossy.
• Symbolic recursion — invoking the model from inside executed code (e.g., a loop), so the number of sub-calls is set by the program, not by the parent’s generated text.
• sub_rlm / sub-call — a function injected into the REPL that runs a fresh LLM (depth-1) on any string slice the code passes it.
• S-NIAH — Single Needle-in-a-Haystack; find one fixed phrase in a long distractor text. Constant complexity in length.
• OOLONG / OOLONG-Pairs — long-context aggregation benchmarks needing every line (linear) or every pair of lines (quadratic); the latter is new in this paper.
• CodeAct / ReAct — agent loops where the model interleaves reasoning with code/tool actions and gets execution feedback.
• BM25 — a classic keyword-based retrieval ranking function used as a retrieval baseline.
• Distillation (trajectory) — fine-tuning a small model on the recorded action sequences of a larger, more capable model.
• Depth-1 recursion — sub-calls are plain LLM calls, not RLMs themselves; the recursion goes exactly one level deep.