RAG vs. GraphRAG: A Systematic Evaluation and Key Insights

TL;DR

Everyone building retrieval pipelines now faces a fork: stick with vanilla RAG (chunk text, embed, retrieve top-k) or invest in GraphRAG (turn text into a knowledge graph / community hierarchy, then retrieve over structure). The hype says GraphRAG is the upgrade. This paper runs the first apples-to-apples benchmark across Question Answering and query-based Summarization, with identical chunking, embeddings, and LLMs, and finds neither dominates. RAG is better at single-hop, detail-heavy queries; GraphRAG (specifically the “local search” community variant) is better at multi-hop reasoning and produces more diverse summaries. The headline practical result: a simple Integration strategy (run both, concatenate the retrieved context, generate once) lifts the best single method by up to 6.4% on multi-hop QA. They also expose a methodological landmine — the LLM-as-a-Judge evaluation that made GraphRAG look great in the original Microsoft paper suffers from severe position bias (flip the order of the two answers and the judge flips its verdict).

Problem & Motivation

The concrete pain: you are choosing a retrieval architecture for a client and the literature is useless for the decision. GraphRAG papers (Microsoft’s community-summarization approach, KG-based LlamaIndex pipelines) each report wins, but on bespoke tasks, hand-picked datasets, and self-graded evaluations. Nobody had put RAG and GraphRAG on the same standard benchmarks with the same knobs and asked “which one, when?”

Why that matters in production:

• GraphRAG is expensive to build. You pay an LLM to extract entities/relations from every chunk, run community detection, and generate community summaries — before you’ve answered a single query. If it doesn’t beat plain RAG on your task, you’ve burned a lot of tokens for nothing.
• The original GraphRAG evidence rests on LLM-as-a-Judge without ground truth, which this paper shows is order-dependent and therefore unreliable.
• “Multi-hop reasoning needs a graph” is folk wisdom, not a measured fact. Before this paper you couldn’t say how much graph structure actually helps, or where it actively hurts.

If you can’t state the pain in one sentence: I don’t know whether the graph is worth the build cost for this client’s query mix, and the existing papers won’t tell me.

What’s New (Core Contribution)

This is an empirical/benchmark paper — the contribution is rigorous measurement plus two practical recipes, not a new model.

1. First controlled RAG-vs-GraphRAG benchmark. Before: scattered, task-specific GraphRAG wins on private datasets with self-grading. Now: one harness, four QA datasets + four summarization datasets, identical chunking (256-token chunks), identical embedding (text-embedding-ada-002), identical generators (Llama-3.1-8B and -70B), graded against ground truth with Precision/Recall/F1, accuracy, ROUGE-2 and BERTScore.
2. A precise map of who wins where. Before: “GraphRAG is the next step.” Now: RAG → single-hop + detail-oriented + comprehensiveness; GraphRAG (Community-Local) → multi-hop reasoning + diversity; GraphRAG (Community-Global) → broad summarization but hallucinates on QA (fails “insufficient information” / Null queries); KG-only GraphRAG → generally underperforms because the extracted graph is incomplete (only ~65% of answer entities even make it into the KG).
3. Two hybrid strategies, measured. Before: no principled way to combine them. Now: Selection (an LLM classifies each query as fact-based→RAG or reasoning-based→GraphRAG; cheap, one pipeline per query) and Integration (run both, concatenate contexts, generate once; more expensive, but +6.4% on MultiHop-RAG/70B).
4. A debunk: position bias in LLM-as-a-Judge. Before: Microsoft’s GraphRAG looked superior under LLM-judged comprehensiveness/diversity. Now: swapping the presentation order of the two summaries flips the judge’s decision — sometimes completely — so that evidence doesn’t establish what it claimed.

How It Works (Technically)

There is no single new algorithm here; the “mechanism” is the evaluation pipeline and the four retrieval variants it compares. Understanding the four variants is the whole game, because each one defines what gets put into the LLM’s context window for a given query — and that context is what makes or breaks the answer.

The shared skeleton. Every method is retrieve(query) -> context, then generate(query, context) -> answer. They differ only in retrieve. Same chunk size, same embedder, same generator across all of them — that’s what makes the comparison fair.

1. RAG (the baseline). Split documents into ~256-token chunks. Embed each chunk with ada-002 into a vector DB. At query time, embed the query, cosine-similarity search, return the top-10 chunks as context. That’s it. No structure, no graph. Single-document tasks build a per-document index; multi-document tasks share one index.

2. KG-GraphRAG (LlamaIndex-style). Indexing: feed each chunk to an LLM that extracts (head, relation, tail) triplets, accumulating a knowledge graph. Retrieval: extract the query’s entities with an LLM, match them to graph nodes, then traverse multi-hop neighbors and collect their triplets. Two flavors — Triplets (return only the relation triples) and Triplets+Text (also return the source text each triple came from). The traversal is what should help multi-hop questions: chaining edges = chaining facts.

3. Community-GraphRAG (Microsoft-style). Build the same LLM-extracted graph, then run community detection to cluster nodes hierarchically, and have an LLM write a summary/report for each community — low-level communities hold detail, high-level ones summarize the low-level ones. Two retrieval modes:

• Local Search → match query entities to the graph, pull entities, relations, descriptions, and low-level community reports. Keeps detail. This is the variant that competes with RAG.
• Global Search → ignore entity matching; semantically retrieve only high-level community summaries. Great for “summarize the whole corpus” questions, terrible for specific facts (it has thrown away the detail), and it hallucinates on Null queries that should return “not enough info.”

The key intuition to internalize: the graph methods trade recall of raw detail for structured connectivity. Global search trades the most detail away; Local search trades the least; KG-only throws detail away and loses anything the extractor missed. Detail-hungry queries punish that trade; reasoning-chain queries reward it.

The two hybrid strategies (the actionable part).

Selection uses in-context learning: give the LLM a few examples of fact-based vs. reasoning-based queries, classify the incoming query, route to RAG or GraphRAG accordingly. One retrieval pipeline runs per query → cheap. Result: +1.1% over the best single method on MultiHop-RAG/70B.

Integration runs both retrievers, concatenates the two contexts, and generates once. Two pipelines per query → ~2x retrieval cost. Result: +6.4% on the same benchmark. The generator effectively gets both “detailed chunks” and “structured neighbors” and sorts it out itself.

Demystifying the evaluation math (no heavy notation in this paper, but two metrics matter):

• ROUGE-2 = fraction of 2-word sequences (bigrams) shared between the generated summary and the reference. Pure lexical overlap. Rewards summaries that reuse the reference’s exact wording → favors detail-faithful RAG.
• BERTScore = cosine similarity between contextual embeddings of generated vs. reference tokens, matched greedily. Semantic, not lexical — gives partial credit for paraphrase. Both are computed as Precision/Recall/F1 (how much of the candidate is justified by the reference, and vice versa).
• Position bias in LLM-as-a-Judge: the authors present the same two summaries to a judge LLM in Order 1 (RAG first) and Order 2 (GraphRAG first). If the verdict depends on order, the judge is responding to position, not quality — so any single-order LLM-judge result is confounded. They show this flips “completely” for RAG vs. GraphRAG-Local.

Architecture & data flow

flowchart TB
  Q[Query] --> R{Retrieval variant}
  T[Text corpus] --> CH[Chunk ~256 tok]
  CH --> EMB[Embed ada-002] --> VDB[(Vector DB)]
  CH --> EXT[LLM triplet extraction] --> KG[(Knowledge graph)]
  KG --> COM[Community detection + LLM reports] --> HC[(Hierarchical communities)]

  R -->|RAG| VDB
  R -->|KG-GraphRAG| KG
  R -->|Community-Local| HC
  R -->|Community-Global| HC

  VDB --> CTX[Context window]
  KG --> CTX
  HC --> CTX
  CTX --> GEN[Llama-3.1 8B/70B] --> A[Answer]

  subgraph Hybrids
    VDB -.Integration: concat.-> CTX
    KG -.Integration: concat.-> CTX
    R -.Selection: LLM routes query.-> R
  end

Schematic: how each retrieval variant fills the context window for a query. Toggle query type (single-hop vs. multi-hop) and watch which method's context contains the answer-bearing pieces. Illustrative, built to match the paper's qualitative findings — not its exact numbers.

The algorithm, simplified

# The two hybrid strategies — the actionable contribution of the paper.
# Stubs: rag_retrieve / graph_retrieve return List[str] context pieces;
#        llm(prompt) -> str is one model call.

def selection(query, k=10):
    # Route each query to ONE retriever based on its nature (cheap: 1 pipeline).
    kind = llm(f"Is this FACT-BASED or REASONING-BASED?\n{query}\n"
               f"Examples: 'When was X born?'=FACT  'How did X cause Y?'=REASONING")
    if "FACT" in kind:
        ctx = rag_retrieve(query, k)          # detail-heavy chunks win single-hop
    else:
        ctx = graph_retrieve(query)            # graph neighbors win multi-hop
    return llm(f"Answer using:\n{ctx}\n\nQ: {query}")

def integration(query, k=10):
    # Run BOTH, concatenate contexts, generate once (2x retrieval cost, +6.4% on MultiHop).
    ctx = rag_retrieve(query, k) + graph_retrieve(query)   # detail AND structure
    return llm(f"Answer using:\n{chr(10).join(ctx)}\n\nQ: {query}")

def graph_retrieve(query):
    # Community-Local: match query entities, pull neighbors + low-level community reports.
    ents = extract_entities(llm, query)
    nodes = match_to_graph(ents)               # entity linking against the built KG
    triples = multi_hop_neighbors(nodes, hops=2)
    reports = low_level_community_reports(nodes)  # keeps detail; Global would drop it
    return triples + reports

Built on Prior Work

Results & Evidence

QA (NQ single-hop, HotPotQA + MultiHop-RAG multi-hop, NovelQA 21 query types):

• RAG tops single-hop NQ (F1 ~64.8 vs. KG-GraphRAG ~34, Community-Local ~63) and detail-oriented NovelQA queries.
• Community-GraphRAG (Local) tops the multi-hop sets (best on HotPotQA and MultiHop-RAG overall).
• Community-Global collapses on Null queries (e.g., 19.3 vs. ~96 for RAG/KG on MultiHop-RAG) — it can’t say “insufficient information” because it only sees high-level summaries → hallucinates.
• The complementarity is the load-bearing finding: on MultiHop-RAG, 13.6% of queries are GraphRAG-only-correct and 11.6% RAG-only-correct. Large non-overlapping pockets of competence justify combining them.

Hybrids: Integration > Selection > best single method, generally. On MultiHop-RAG/70B: Selection +1.1%, Integration +6.4%. The tradeoff is explicit — Selection is 1x cost, Integration is ~2x.

Summarization (SQuALITY, QMSum, ODSum-story, ODSum-meeting; ROUGE-2 + BERTScore vs. ground truth):

• RAG generally wins, especially multi-document. KG-GraphRAG improves a lot when you add source text to triplets. Integration ≈ RAG (it’s dominated by the detail RAG already supplies).
• Under LLM-as-a-Judge: RAG wins “comprehensiveness,” GraphRAG-Global wins “diversity” — but Local-vs-RAG judgments flip entirely with order, exposing position bias.

What the evidence does NOT establish (read this before quoting it to a client):

• Only two generators, both Llama-3.1. No GPT-4-class or reasoning models — the gap could shrink or shift with a stronger generator.
• LLM-built graphs only. Every GraphRAG result is hostage to LLM extraction quality (the 65% coverage problem). A better extractor or a curated KG could change the verdict — the paper says so in its limitations.
• QA + query-based summarization only. No agentic/tool-use tasks, no code, no truly corpus-scale “global sensemaking” where Global search is designed to shine (their datasets are role/event-specific, which structurally disadvantages Global).
• Multi-hop “win” for GraphRAG is modest — Community-Local is roughly comparable to RAG overall, not a blowout. The honest read: graph structure buys you a specific slice of queries, not a tier change.

How You’d Use It

You run an AI services company; this paper is a sales-engineering decision tool, not a model to ship.

• Default to RAG, justify the graph. For most client query mixes (lookup, fact retrieval, detail-heavy support), plain top-k RAG is the strong, cheap baseline. Don’t sell a GraphRAG build unless the client’s queries are genuinely multi-hop/relational. This paper is your evidence to not over-engineer.
• Sell “Integration” as a premium tier. The +6.4% on multi-hop is a real, defensible upsell: “we run both retrievers and merge context.” It’s simple to implement (concatenate two retrieval calls) and the cost story is honest (~2x retrieval, same generation). Great for high-stakes QA where accuracy beats cost.
• Build the Selection router for cost-sensitive clients. An LLM query classifier in front of two retrievers gives most of the benefit at 1x cost — a clean middle tier.
• In a multi-agent system, treat RAG and GraphRAG as two retriever tools and let an orchestrator agent pick (that’s exactly Selection) or fan out and merge (Integration). This drops straight into an ARC-MAS-style setup: a router agent + two retrieval workers + a synthesis step.
• Use the position-bias finding to set evaluation standards. When you A/B retrieval approaches for clients, never trust a single-order LLM-judge. Randomize order, or use ground-truth metrics. This is a credibility differentiator vs. shops that quote LLM-judge numbers naively.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value — a working RAG-vs-GraphRAG bench plus the Integration hybrid:

1. RAG arm (½ day). LlamaIndex or LangChain: load docs → chunk (256 tok) → embed (any decent embedder; ada-002 or open bge/nomic) → vector store (FAISS/Chroma) → top-10 retriever → LLM generate.
2. GraphRAG arm (1–2 days, this is the hard part). Use Microsoft GraphRAG for the Community variant (it does extraction + community detection + reports for you) or LlamaIndex KnowledgeGraphIndex for the KG variant. The hard, expensive step is graph construction — it’s a per-chunk LLM call and it’s where quality leaks in (watch the answer-entity coverage).
3. Integration hybrid (1 hour). context = rag_retrieve(q) + graph_retrieve(q); dedupe; generate once. This is the cheapest path to the headline win.
4. Selection router (½ day). A few-shot classifier prompt (fact vs. reasoning) → route. Cache classifications.
5. Honest eval harness (1 day). Hold out QA pairs; compute F1/accuracy against ground truth. For summaries, ROUGE-2 + BERTScore (HuggingFace evaluate). If you must use an LLM judge, run both orders and average — bake the position-bias fix in from day one.

The two genuinely hard parts: (a) graph construction cost/quality, and (b) building a fair harness so the comparison means something (identical chunking/embedding/generator — easy to get wrong and accidentally rig).

How to Improve It

Limitations are your roadmap:

1. Better graph construction. The whole GraphRAG case is bottlenecked by ~65% answer-entity coverage. Swap LLM triplet extraction for a fine-tuned IE model, add an entity-completion pass, or do iterative extraction (re-extract on retrieval misses). Measure coverage as a first-class metric — it predicts QA accuracy.
2. Learned routing, not few-shot. Selection uses an in-context classifier. Train a small classifier (or a bandit) on actual win/loss per query type to route — and let it choose RAG / GraphRAG / both (cost-aware Integration only when it pays).
3. Smarter merge than concatenation. Integration just concatenates contexts. Try reranking the union, or a reciprocal-rank-fusion over the two retrievers’ results, to fit more signal in a fixed context budget.
4. Stronger / reasoning generators. Re-run with a GPT-4-class or explicit reasoning model. Hypothesis: a better generator narrows GraphRAG’s multi-hop edge (it can chain facts from raw chunks itself), shifting the build-vs-buy math.
5. Test where Global is supposed to win. Their datasets are role/event-specific, which hobbles Global search. Add a true corpus-level “sensemaking” benchmark to fairly evaluate the community-summary approach — and pair it with a bias-corrected, ground-truth-anchored eval.
6. Cost-quality Pareto curve. None of the results report tokens/latency. Add a $/query axis; “Integration is +6.4% at 2x cost” is the decision a client actually makes.

Glossary

• RAG (Retrieval-Augmented Generation) — retrieve relevant text, stuff it in the prompt, then generate; grounds the LLM in external data.
• GraphRAG — RAG where retrieval happens over a graph (knowledge graph or community hierarchy) instead of (or in addition to) flat text chunks.
• Chunk — a fixed-size slice of a document (~256 tokens here) that is the unit of embedding and retrieval.
• Embedding / vector DB — a numeric vector representing text meaning; a store that finds nearest vectors for similarity search.
• Top-k retrieval — return the k most similar chunks to the query (k=10 here).
• Knowledge graph (KG) / triplet — facts as nodes and edges; a triplet is (head, relation, tail), e.g. (Einstein, born_in, Ulm).
• Multi-hop query — a question whose answer requires chaining several facts (hops) together.
• Single-hop query — answerable from one fact / one passage.
• Community detection — graph clustering that groups densely connected nodes; here each cluster gets an LLM-written summary.
• Local vs. Global search (Community-GraphRAG) — Local pulls entity-matched detail + low-level community reports; Global pulls only high-level community summaries.
• ROUGE-2 — overlap of 2-grams between generated and reference text (lexical similarity metric).
• BERTScore — similarity of contextual embeddings between generated and reference text (semantic similarity metric).
• LLM-as-a-Judge — using an LLM to score/compare outputs instead of human or ground-truth grading.
• Position bias — a judge LLM’s tendency to prefer an answer based on where it appears in the prompt, not its quality.
• In-context learning (ICL) — getting an LLM to do a task from a few examples in the prompt, with no weight updates.
• Selection / Integration — the paper’s two hybrids: route each query to one retriever (Selection) vs. run both and merge contexts (Integration).
• Null query — a question that should be answered “insufficient information”; a hallucination trap.