Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks

TL;DR

Naive RAG (“embed query → top-k similarity → stuff chunks into the prompt”) falls apart on real queries: shallow matching, noise, and a rigid linear pipeline that can’t adapt. This paper’s contribution is not a new algorithm — it’s an organizing framework. It decomposes every RAG technique in the literature into a three-tier hierarchy (Modules → Sub-modules → Operators), then shows that any RAG system — from vanilla to Self-RAG to FLARE — is just a computational graph (a “RAG Flow”) built from those bricks. It then catalogs the six recurring flow patterns (linear, conditional, branching, looping/iterative/recursive/adaptive, plus tuning) that the whole field keeps reinventing. The value is conceptual leverage: a vocabulary and a mental model that lets you design, compare, debug, and pitch RAG architectures systematically instead of ad hoc.

Problem & Motivation

If you’ve shipped RAG for a client, you’ve felt this pain. The textbook pipeline is: embed the user’s question, pull the top-k most cosine-similar chunks, paste them into the prompt, generate. It works in demos and breaks in production for two structural reasons the paper names directly:

1. Shallow understanding of queries. Semantic similarity between a question and an answer chunk is often weak. “What were the side effects in the 2021 trial?” doesn’t embed near the paragraph that actually contains them. Similarity is a blunt instrument.
2. Retrieval redundancy and noise. Dumping all k chunks into context actively hurts — LLMs lose the signal in the noise, forget the middle of long contexts (“lost in the middle”), and hallucinate more, not less.

The field responded with a flood of fixes — query rewriting, HyDE, reranking, compression, iterative retrieval, self-reflective retrieval, knowledge-graph indexing, fine-tuned retrievers. But each arrived as a bespoke pipeline with its own diagram. There was no shared language. You couldn’t say “Self-RAG is FLARE plus a fine-tuned judge minus the confidence gate” because nobody had defined the primitives. The paradigm (“retrieve-then-generate”) had fallen behind the practice. The result: every team rebuilds the same orchestration logic, can’t compare approaches apples-to-apples, and can’t tell a client why their architecture is the right one.

The concrete pain in one sentence: RAG had outgrown its own definition, leaving practitioners with a pile of incompatible techniques and no framework to assemble, compare, or maintain them.

What’s New (Core Contribution)

This is a survey/architecture paper, so the novelty is in the organization, not a benchmark win. Three genuine contributions:

• A three-tier decomposition (Modules → Sub-modules → Operators).

  • Before: RAG described loosely as “indexing, retrieval, generation” with a grab-bag of add-ons.
  • Now: A strict hierarchy. L1 Modules are the six stages (Indexing, Pre-retrieval, Retrieval, Post-retrieval, Generation, Orchestration). L2 Sub-modules refine each (e.g., Pre-retrieval contains Query Expansion, Query Transformation, Query Construction). L3 Operators are the concrete, swappable implementations (e.g., HyDE, Step-back, Multi-Query are operators inside Query Transformation). This is the “LEGO brick” insight: operators are interchangeable parts with defined sockets.

• RAG as a computational graph (“RAG Flow”).

  • Before: RAG drawn as a linear arrow chain.
  • Now: G = (V, E) — modules are nodes, control/data flow are edges. This formalizes that RAG can branch, loop, and route. Crucially, it makes the Orchestration module (Routing, Scheduling, Fusion) a first-class citizen, not an afterthought. Orchestration is what separates Modular RAG from its ancestors.

• Six recurring flow patterns.

  • Before: Every paper’s pipeline looked unique.
  • Now: The authors show the whole field collapses into a small set of reusable patterns: Linear, Conditional, Branching, Loop (iterative / recursive / adaptive), and Tuning. Each comes with a precise algorithm and a real-world exemplar (RRR, REPLUG, ITER-RETGEN, ToC, FLARE, Self-RAG, RA-DIT). This is the most actionable part — it’s a design pattern catalog for RAG.

The honest read: there’s no new model, no SOTA number, no released system. The contribution is a taxonomy and a shared vocabulary. That sounds modest, but it’s the kind of paper that changes how a field talks — analogous to “Gang of Four” design patterns for OOP.

How It Works (Technically)

The framework rests on a simple formalization. A Modular RAG system is the tuple:

G = {q, D, M, {Ms}, {Op}}

• q = the query, D = the document repository (chunks d_i).
• M = modules, {Ms} = sub-modules, {Op} = operators.

A RAG Flow is an ordered arrangement of parameterized modules: F = (M_φ1, ..., M_φn), decomposable into a graph of sub-functions. In the trivial case it’s a linear chain; in general it’s a directed graph that can branch and loop.

Let’s demystify the key equations so the math earns its place.

Retrieval (Eq. 2–3). R : top-k Sim(q, d_i) → Dq. Operationally: embed the query, score every chunk by similarity, keep the top k. Sim is dot product or cosine: Sim(q, d_i) = e_q · e_d / (||e_q|| ||e_d||). Nothing exotic — this is the one brick everyone already knows. The paper’s point is that this is one operator in one sub-module, not the whole system.

Generation (Eq. 4). y = LLM([Dq, q]) — concatenate retrieved chunks and query, generate. Again, one operator.

Routing (Eq. 18–22) — this is where Modular RAG diverges. A routing function fr : Q → F maps a query to a whole flow. Two flavors:

• Metadata routing (Eq. 19): score_key(q_i, F_j) = (1/|K'_j|)·|K_i ∩ K'_j| — count how many of a flow’s predefined keywords appear in the query, normalized. Plain English: “does this query mention the keywords that flow F is built for?” Highest score wins (Eq. 20, argmax).
• Semantic routing (Eq. 21): instead of keyword overlap, classify the query’s intent with an LLM (P_Θ(θ|q) = softmax over intent log-probs), then map intent → flow via δ(·). Plain English: “what is this query about, and which pipeline handles that topic?”
• Hybrid (Eq. 22): α_i = a·score_key + (1−a)·score_semantic — a weighted blend so keyword precision and semantic recall complement each other.

Scheduling — the loop controller. This is the “Judge” function that decides, after a generation step, whether to stop or retrieve again. Two implementations worth understanding:

• Rule judge (token-confidence gate): accept the tentative answer ŝ_t only if every token’s probability ≥ threshold τ; otherwise regenerate with fresh retrieval. This is exactly FLARE’s trick — low-confidence tokens are a signal that the model is guessing and needs more evidence.
• LLM judge: either prompt the LLM to decide in-context (no fine-tuning, but format-fragile), or fine-tune it to emit special control tokens (the Toolformer/Self-RAG approach) that directly trigger retrieve/critique actions.

Fusion — merging parallel branches. When you fan out into multiple sub-queries or pipelines, you need to merge. Three operators:

• LLM fusion: just ask an LLM to synthesize the branch outputs (summarize-then-merge if it overflows context).
• Weighted ensemble (Eq. 23–24): p(y|q,Dq) = Σ_d p(y|d,q)·λ(d,q) where the weight λ(d,q) = softmax(s(d,q)) is the normalized similarity. Plain English: each retrieved doc votes on the next token, weighted by how relevant it is. This is REPLUG’s mechanism — it ensembles at the token-probability level, not the text level.
• RRF (Reciprocal Rank Fusion): merge multiple ranked lists by summing reciprocal ranks. Robust when sources are heterogeneous (e.g., combining a sparse BM25 list with a dense vector list).

The RL bits (for the Tuning pattern). Several operators are trained, not prompted:

• Contrastive retriever fine-tuning (Eq. 10): the standard InfoNCE loss — pull the query embedding toward the positive doc d+, push it away from negatives d−. This is how you adapt a retriever to medical/legal jargon.
• LM-Supervised Retrieval (Eq. 11): instead of human labels, use the LLM’s likelihood of the right answer given a doc, P_LM(y|d,q), as the training signal for the retriever. The LLM tells the retriever “this doc helped me answer correctly.”
• Dual fine-tuning / RA-DIT (Eq. 15–16): train retriever and generator together, aligning the retriever’s relevance distribution P_R(d|q) with the LLM’s preference distribution via KL divergence loss. KL divergence here measures how far the retriever’s “what’s relevant” distribution is from the generator’s “what actually helped” distribution; minimizing it makes the two agree. RRR (Rewrite-Retrieve-Read) goes further and treats query rewriting as a Markov decision process, using the final LLM answer quality as the reward to RL-train the rewriter — the rewriter learns to phrase queries the way the retriever likes.

Architecture & data flow

flowchart TB
  Q[Query q] --> ORCH{Orchestration:<br/>Routing fr}
  ORCH -->|flow A| PRE[Pre-retrieval<br/>expand / rewrite / HyDE]
  ORCH -->|flow B| PRE
  PRE --> RET[Retrieval<br/>sparse / dense / hybrid]
  RET --> POST[Post-retrieval<br/>rerank / compress / select]
  POST --> GEN[Generation<br/>LLM + optional verify]
  GEN --> JUDGE{Scheduling:<br/>Judge — done?}
  JUDGE -->|no, retrieve again| PRE
  JUDGE -->|branches to fuse| FUSE[Fusion<br/>RRF / weighted / LLM]
  FUSE --> GEN
  JUDGE -->|yes| Y[Answer y]
  IDX[(Indexing<br/>chunking / KG / hierarchical)] -.feeds.-> RET

The three things that make this graph modular rather than linear: Routing (pick the flow at the top), Scheduling (the Judge loop-back edge), and Fusion (merge parallel branches). Strip all three out and you’re back to Naive RAG — which the paper explicitly frames as a degenerate special case.

Interactive RAG Flow builder (schematic). Toggle modules on/off and switch the pattern (linear / conditional / branching / loop) to see how the same six bricks reconfigure into different architectures. This is the "LEGO" claim made literal.

The flow patterns, side by side

The catalog is the practical heart. Each pattern is a shape the graph takes:

flowchart LR
  subgraph Linear
    L1[M1]-->L2[M2]-->L3[M3]
  end
  subgraph Conditional
    C0[router]-->|cond A|CA[flow A]
    C0-->|cond B|CB[flow B]
  end
  subgraph Branching
    B0[split]-->BA[branch 1]
    B0-->BB[branch 2]
    BA-->BM[merge]
    BB-->BM
  end
  subgraph Loop
    P0[retrieve]-->P1[generate]-->P2{judge}
    P2-->|again|P0
    P2-->|stop|PE[done]
  end

• Linear (e.g., RRR): fixed sequence pre → retrieve → post → generate.
• Conditional (router picks one pipeline): different flows for political vs. entertainment vs. technical queries — different sources, models, prompts, tolerances.
• Branching (run several in parallel, then merge): pre-retrieval branching expands one query into many sub-queries (Multi-Query); post-retrieval branching retrieves once but generates per-chunk and ensembles (REPLUG).
• Loop — the most powerful, three sub-types:
  • Iterative (ITER-RETGEN): fixed number of retrieve→generate rounds, each round using the last output to retrieve better.
  • Recursive (ToC): tree-structured, each step transforms the query and deepens; clear termination (max depth/nodes).
  • Adaptive/Active (FLARE, Self-RAG): the LLM itself decides when to retrieve and when to stop — the closest thing to an agent. FLARE uses token-confidence; Self-RAG uses fine-tuned control tokens.

The algorithm, simplified

The single idea that captures Modular RAG: execute a graph of operators, with a router choosing the flow and a judge controlling the loop. Here’s the adaptive (active) loop — the pattern that turns RAG into something agent-like:

# Modular RAG: adaptive flow. Operators are swappable; the loop is the contribution.
# Stubs: route(q)->flow, retrieve(q)->chunks, llm(prompt)->text, confident(text)->bool

def modular_rag(q, max_steps=4):
    flow = route(q)                 # Orchestration/Routing: pick pipeline for THIS query
    history, answer = [], None
    for step in range(max_steps):
        sub_q = transform(q, history)        # Pre-retrieval: rewrite/expand using context so far
        chunks = flow.retrieve(sub_q)        # Retrieval: sparse/dense/hybrid — whichever the flow uses
        chunks = flow.post(sub_q, chunks)    # Post-retrieval: rerank -> compress -> drop irrelevant
        answer = llm([history, sub_q, chunks])  # Generation
        history.append(answer)
        # Scheduling/Judge: the heart of "loop" patterns.
        # FLARE-style: stop only if every token cleared the confidence bar.
        if confident(answer):                # else: another retrieval round with a better query
            break
    return synthesize(history)               # Fusion: merge multi-step / multi-branch outputs

Swap route for a constant and confident for True and you’ve written Naive RAG in the same skeleton — which is exactly the paper’s claim that older paradigms are special cases.

Built on Prior Work

The paper is a synthesis; its job is to place prior work, not beat it. The lineage it organizes:

The conceptual ancestor outside RAG is software design patterns (Gang of Four) and modular/microservice architecture — the paper explicitly invokes modularization as the trend it’s importing into RAG.

Results & Evidence

Be clear-eyed here: this paper has no experiments, no benchmark table, no ablations. It’s a position/survey paper. That’s not a flaw — it’s the genre — but it changes what “evidence” means.

The evidence offered is coverage and explanatory power: the authors demonstrate that a wide range of named systems (RRR, REPLUG, ITER-RETGEN, ToC, FLARE, Self-RAG, RA-DIT, DR-RAG, PlanRAG, and more) each map cleanly onto one of their patterns. The implicit claim is “our taxonomy is complete enough that every technique you care about fits.”

What the evidence does establish:

• A coherent, reasonably exhaustive vocabulary for RAG components and topologies.
• That the field’s “novel pipelines” are mostly recombinations of a small set of primitives.

What it does NOT establish:

• That any particular flow pattern is better than another — no head-to-head numbers.
• When to choose conditional vs. branching vs. loop — guidance is qualitative (“depends on the scenario”).
• That the taxonomy is the only or optimal carving — it’s one sensible decomposition, not a proven-minimal one.
• Cost/latency tradeoffs — looping and branching multiply LLM calls, and the paper doesn’t quantify the bill.

Treat it as a map, not a leaderboard. Its usefulness is in design and communication, not in telling you which model wins.

How You’d Use It

For an AI services company, this paper is unusually high-leverage precisely because it’s a framework, not a model. Concrete uses:

1. A shared design language with clients and engineers. When scoping a RAG build, you can say “this is a conditional flow routing to two branches, with a token-confidence loop on the technical branch.” That’s a spec, an estimate, and a debugging map in one sentence. It turns RAG from artisanal to architectural.

2. A diagnostic checklist. When a client’s RAG underperforms, walk the six modules: Is indexing the bottleneck (bad chunking)? Pre-retrieval (raw query too vague — add rewrite/HyDE)? Retrieval (wrong retriever — add hybrid)? Post-retrieval (noise — add rerank/compress)? Generation (weak model)? Orchestration (no routing — one-size pipeline)? Most failures localize to one module. This is a productizable audit.

3. A reference architecture catalog. Pre-build the six patterns as templates in LangGraph/LlamaIndex. Each new client engagement becomes “pick a pattern, swap operators,” not “design from scratch.” That’s margin.

4. The MAS connection. You’ve built multi-agent systems. The Adaptive/Active loop is an agent: route = role selection, scheduling/Judge = the control policy, operators = tools. Modular RAG gives you a principled way to fold retrieval into your agent graphs — retrieval becomes just another tool node, and routing becomes agent handoff. This collapses the artificial wall between “RAG systems” and “agent systems.”

5. Sales and differentiation. Most vendors ship Naive RAG and call it done. Being able to articulate (and build) routing/branching/looping flows is a credible moat and a clear upsell ladder: start linear, add reranking, add routing, add adaptive loops as the client’s needs grow.

Build Your Own (Minimal Recipe)

To get ~80% of the value, don’t build all six patterns. Build the skeleton that makes flows swappable, then implement two patterns.

Components (build order):

1. Operator registry. A dict mapping names → callables with a uniform signature, e.g. op(state) -> state. State is a dict carrying query, chunks, answer, history. This is the LEGO socket — every brick reads/writes the same shape.
2. Indexing + Retrieval baseline. Chunk docs, embed (any off-the-shelf embedder), store in a vector DB (FAISS/Chroma/pgvector). Add BM25 for hybrid. This is the one piece that’s mostly solved — don’t over-invest.
3. Pre/Post operators. Two pre (rewrite, HyDE — both are just LLM prompts) and two post (a cross-encoder reranker like bge-reranker, and an LLM-critique filter). Each is <30 lines.
4. Orchestration. A router (start with semantic: an LLM intent classifier → flow name) and a Judge (start with FLARE-style token confidence or a simple “is this answer complete? yes/no” LLM call).
5. Graph runner. Use LangGraph — it’s literally a state-graph executor, which is exactly the G=(V,E) abstraction. Define nodes = operators, edges = flow, conditional edges = router/judge. You get looping and branching for free.

Implement two patterns first: Linear (rewrite → hybrid retrieve → rerank → generate) for the 80% case, and Adaptive Loop (the code above) for the hard multi-hop case. Add Conditional routing between them.

The two genuinely hard parts:

• The Judge / stopping criterion. Token-confidence requires logprobs (not all APIs expose them); LLM-as-judge is reliable-ish but adds latency and cost per loop. Getting this to terminate well — not too early, not infinite — is the real engineering, not the retrieval.
• Fusion quality. Merging branches without contradiction or bloat is subtle. RRF is cheap and robust for ranked lists; LLM fusion is better but can hallucinate during synthesis. Start with RRF.

Reach for: LangGraph (orchestration), LlamaIndex (indexing/retriever zoo, has many of these operators prebuilt), bge/gte embedders, bge-reranker or Cohere Rerank, FAISS/pgvector.

How to Improve It

Limitations are leverage. Five concrete, testable directions:

1. Add a cost/latency dimension to the taxonomy. The paper is silent on the bill. Each pattern has a call-count profile (linear = O(1), branching = O(branches), loop = O(steps)). Build a model that, given accuracy targets and a budget, recommends a pattern. That’s a genuinely useful and saleable tool the paper doesn’t provide.

2. Learn the router instead of hand-defining it. Routing is currently keyword/intent rules. Train a small classifier (or RL policy) on (query, best-flow, outcome) tuples to learn which flow maximizes answer quality per dollar. This turns Orchestration from configuration into a learned policy — and it’s testable against a held-out QA set.

3. Auto-construct flows (Neural Architecture Search for RAG). Since flows are graphs of typed operators, you can search the space. Given a dataset, automatically assemble and evaluate candidate flows. The paper hands you the search space; nobody’s automated the search.

4. Unify with agent frameworks explicitly. The adaptive loop is an agent but the paper keeps them separate. Define the formal mapping (operator ↔ tool, judge ↔ policy, fusion ↔ aggregation) and you get one execution model for RAG + tools + multi-agent. Then test whether agent-style planning beats fixed flow patterns on multi-hop benchmarks.

5. Add observability/eval as a first-class module. The paper lists Indexing→Orchestration but omits Evaluation/Monitoring, which is where production RAG actually lives or dies. A seventh module — per-operator tracing, faithfulness/relevance scoring, regression gates — would make the framework deployable, not just describable. This is directly productizable as a client offering.

Glossary

• RAG (Retrieval-Augmented Generation) — feeding an LLM relevant external documents at inference time so it answers from data, not just memory.
• Naive RAG — the baseline: embed query, top-k similarity search, stuff chunks into the prompt, generate. No routing or loops.
• Operator — the smallest swappable unit (L3); a concrete implementation like HyDE or BM25. The “LEGO brick.”
• Module / Sub-module — L1 stages (Indexing, Pre-retrieval, …) and their L2 refinements; the brick categories.
• RAG Flow — a specific wiring of modules/operators into a computational graph G=(V,E) for one system.
• Flow pattern — a recurring graph shape: linear, conditional, branching, loop, tuning.
• Orchestration — the control layer: Routing (pick the flow), Scheduling (loop control via a Judge), Fusion (merge branches).
• Routing — mapping a query to the right pipeline, via keyword overlap (metadata) or LLM intent (semantic).
• Scheduling / Judge — the decision after each step: stop, or retrieve/generate again. Drives loop patterns.
• Fusion — merging parallel branch outputs: LLM synthesis, weighted token ensemble, or Reciprocal Rank Fusion (RRF).
• HyDE — generate a hypothetical answer and retrieve against that (answer-to-answer similarity beats question-to-answer).
• Step-back prompting — abstract the query into a higher-level concept question, retrieve on both.
• Rerank — reorder retrieved chunks by a stronger relevance model (cross-encoder) before generation.
• Sparse vs. Dense vs. Hybrid retriever — keyword stats (BM25), neural embeddings (BGE), or both combined.
• Iterative / Recursive / Adaptive retrieval — loop variants: fixed rounds / query-deepening tree / LLM decides when to retrieve and stop.
• FLARE — adaptive RAG that retrieves only when the next sentence contains low-confidence tokens.
• Self-RAG — fine-tunes the LLM to emit control tokens (Retrieve, critique, utility) that steer the RAG process.
• REPLUG — ensembles per-document next-token probabilities; classic post-retrieval branching.
• RA-DIT — jointly fine-tunes retriever and generator, aligning their preferences via KL divergence.
• RRR (Rewrite-Retrieve-Read) — RL-trains a query rewriter using the final answer quality as reward (query rewriting as an MDP).
• Contrastive loss (InfoNCE) — training objective that pulls a query toward relevant docs and pushes it from irrelevant ones; used to fine-tune retrievers.
• KL divergence — a measure of how different two probability distributions are; minimizing it makes the retriever’s “relevant” distribution match the generator’s “helpful” distribution.
• MDP (Markov Decision Process) — the RL formalism (states, actions, rewards) used to frame query rewriting as a learnable decision policy.
• RRF (Reciprocal Rank Fusion) — merge several ranked lists by summing reciprocal ranks; robust across heterogeneous sources.
• Lost in the middle — LLMs attend to the start/end of long contexts and neglect the middle; motivates reranking/compression.