From RAG to Memory: Non-Parametric Continual Learning for Large Language Models

TL;DR

RAG is how we give LLMs new knowledge without retraining them, but plain vector search retrieves passages in isolation — it can’t “connect the dots” across documents (associativity) or understand a long, sprawling narrative (sense-making). Earlier attempts to fix this with knowledge graphs and summaries did improve those hard cases, but quietly got worse at the easy factual questions that vanilla RAG already nails. HippoRAG 2 is the first structure-augmented RAG that wins across all three — factual, multi-hop, and discourse — by indexing both concepts (graph nodes) and full passages into one graph, linking the whole query to graph triples instead of bare entities, using an LLM as a “recognition memory” filter, and running Personalized PageRank to spread relevance through the graph. The payoff: a 7-point average gain on associative (multi-hop) tasks over the best embedding model, with zero regression on simple QA.

Problem & Motivation

Here’s the concrete pain. You run an AI services company and a client says “the model needs to know our internal docs, and the docs change weekly.” You have three options to teach an LLM new knowledge:

1. Continual fine-tuning — retrain on new data. Expensive, and it suffers catastrophic forgetting (learning new facts erases old ones).
2. Model editing — surgically patch specific weights. The edits are too localized; related facts that should change don’t.
3. RAG — leave the model frozen, retrieve relevant text at query time, stuff it into the prompt. Cheap, robust, no forgetting. This is why RAG is the de facto answer in production.

But standard RAG retrieves with a single vector-similarity lookup: embed the query, find the top-k most similar passages, done. That works great when the answer lives in one passage (“What year was X founded?”). It falls apart on two things humans do effortlessly:

• Associativity — multi-hop reasoning. “Who directed the movie that won Best Picture the year the director of Jaws turned 30?” The answer requires chaining facts that live in different passages. Independent vector lookups never see the chain.
• Sense-making — understanding a large, complex body of text (a whole novel, a sprawling incident log) where the answer isn’t a snippet but a synthesis.

The frustrating part the paper highlights: the field’s fixes for these — GraphRAG, RAPTOR, LightRAG, the original HippoRAG — each won on their own benchmark but regressed below plain RAG on simple factual QA. RAPTOR drops to 50.7 F1 on NQ vs. 61.9 for the best embedding model; the original HippoRAG collapses to 2.4 on long-context discourse. They traded breadth for depth. A real “memory system” can’t have that tradeoff — it has to handle the easy and the hard.

What’s New (Core Contribution)

HippoRAG 2 is the original HippoRAG’s machinery (open knowledge graph + Personalized PageRank) with three targeted upgrades, each fixing a specific way the v1 entity-centric design lost context:

• Dense–sparse integration (passage nodes in the graph). Before: the graph contained only phrase/concept nodes (sparse, lossy); passages were scored separately and the two scores were just summed. Now: every passage is also a node in the same graph, connected by “contains” edges to the phrases extracted from it. Concepts and their original context live in one structure, so graph search can flow between them.
• Deeper contextualization (query-to-triple linking). Before: the query was parsed with Named Entity Recognition, and the bare entities were matched to graph nodes — concept-centric, throwing away the query’s context. Now: the entire query is embedding-matched against triples (subject–relation–object facts), which carry relational context. This single change is the biggest driver: it lifts multi-hop recall from 74.6 to 87.1.
• Recognition memory (LLM triple filter). Before: retrieved triples were used as-is. Now: the top-k retrieved triples are passed to an LLM that filters out irrelevant ones before they seed the graph search — mimicking the human distinction between recall (free retrieval) and recognition (judging whether a cue is relevant).

The genuinely new bit is the integration: putting passages and phrases in one graph and seeding PageRank from query-matched, LLM-filtered triples plus all passage nodes. The PageRank engine and the OpenIE-built graph are inherited from v1 — that’s repackaged, and the paper is honest about it.

How It Works (Technically)

There are two phases: offline indexing (build the memory once) and online retrieval (answer a query). Think of the graph as the hippocampus, the embedding model as the brain region that links memories, and the LLM as the neocortex that reasons.

Offline indexing — building the graph

1. OpenIE by LLM. Feed each passage to an LLM and ask it to extract open knowledge-graph triples: (subject, relation, object), e.g. (Steven Spielberg, directed, Jaws). Subjects and objects become phrase nodes; relations become relation edges. “Open” means no fixed schema — the LLM invents whatever relations fit.
2. Synonym detection by embedding. Embed every phrase node; if two phrases have cosine similarity above a threshold (e.g. “NYC” and “New York City”), add a synonym edge. This is what lets new documents merge with old knowledge instead of sitting in a silo — the continual-learning glue.
3. Dense–sparse integration. Add a passage node for each original passage, with a context edge (“contains”) to every phrase extracted from it. Now the graph holds both the lossy-but-general concepts (sparse coding) and the rich original context (dense coding).

Online retrieval — answering a query

1. Retrieve passages and triples. Embed the query. Score it against passage embeddings (→ candidate passages) and against triple embeddings (→ candidate triples). This is the query-to-triple step: matching the whole query against relational facts rather than matching extracted entities against nodes.
2. Recognition memory (triple filtering). Hand the top-k triples to an LLM: “which of these are actually relevant to the query?” Keep the survivors T' ⊆ T. This strips noise before it can mislead the graph search.
3. Assign seed node weights. From the filtered triples, pick up to k phrase nodes (weighted by their ranking across the triples that mention them). Also take all passage nodes as seeds (broad activation helps multi-hop). Phrase nodes get reset probability from their ranking scores; passage nodes get reset probability from their embedding similarity, scaled by a weight factor (0.05) that keeps phrases dominant.
4. Personalized PageRank (PPR) graph search. Run PageRank, but instead of restarting from a uniform distribution, restart from those seed nodes. Relevance “flows” out from the seeds along edges — a passage two hops from a strong seed gets boosted even if its own embedding didn’t match the query. This is the mechanism that does multi-hop reasoning.
5. QA reading. Take the top-ranked passages by PageRank score, stuff them into the LLM, generate the answer.

Demystifying Personalized PageRank — the one piece of math that matters. Classic PageRank computes the importance of every node as the stationary distribution of a random walk: imagine a surfer hopping along edges forever; the fraction of time spent on each node is its score. Personalized PageRank adds a twist: at each step, with some probability the surfer teleports back — not to a random node, but to a chosen set of seed nodes (a “personalization vector”). The update is iterated until it converges:

r ← (1 − α) · M · r  +  α · s

In plain English: r is the vector of node scores you’re solving for. M is the graph’s transition matrix (where the walk can step). s is the seed/personalization vector — your query’s entry points, mostly the filtered triple phrases. α is the teleport probability (how strongly you stay anchored to the seeds vs. wander the graph). Operationally: it’s “spreading activation” — start energy at the query-relevant nodes and let it diffuse through the graph, so nodes structurally close to your query (even multiple hops away) light up. That diffusion is the associative memory. The whole novelty of HippoRAG over plain RAG is replacing “top-k cosine similarity” with “where does activation settle after it flows through the knowledge graph.”

Architecture & data flow

flowchart TB
  subgraph Offline["Offline Indexing (build memory once)"]
    P[Passages] -->|OpenIE by LLM| T[Triples: subj-rel-obj]
    T --> PN[Phrase nodes + relation edges]
    PN -->|embed + threshold| SY[Synonym edges]
    P -->|passage nodes + 'contains' edges| KG[(Open Knowledge Graph)]
    PN --> KG
    SY --> KG
  end
  subgraph Online["Online Retrieval & QA (per query)"]
    Q[Query] -->|embed| RT[Match query to triples and passages]
    RT --> RM[Recognition memory: LLM filters triples]
    RM --> SEED[Seed nodes + reset weights]
    SEED --> PPR[Personalized PageRank over KG]
    KG -. graph .-> PPR
    PPR --> RANK[Passages ranked by PageRank]
    RANK --> QA[LLM reads top passages -> Answer]
  end

Schematic of Personalized PageRank as spreading activation. The query seeds two nodes (bright); click "step" to watch relevance diffuse along edges and light up structurally-connected passages a vanilla cosine lookup would miss. Illustrative, not the paper's exact graph.

The algorithm, simplified

# HippoRAG 2 online retrieval — the core loop (model/embed calls stubbed)
def retrieve(query, kg, embed, llm, k=5, alpha=0.5, passage_weight=0.05):
    # 1. Query-to-triple + query-to-passage linking (the "deeper contextualization")
    cand_triples  = top_k(embed(query), kg.triple_embeddings, k)      # relational, not bare entities
    cand_passages = top_k(embed(query), kg.passage_embeddings, k)

    # 2. Recognition memory: LLM keeps only triples actually relevant to the query
    good_triples = llm(f"Keep triples relevant to: {query}\n{cand_triples}")  # -> subset

    if not good_triples:                       # fall back to plain dense retrieval
        return cand_passages

    # 3. Seed the personalization vector: phrase nodes from filtered triples + ALL passages
    seeds = {}
    for ph in phrase_nodes(good_triples):
        seeds[ph] = rank_score(ph, good_triples)          # phrases dominate
    for pg, sim in cand_passages:
        seeds[pg] = passage_weight * sim                  # passages nudge, weighted down

    # 4. Personalized PageRank: relevance flows from seeds through the graph (multi-hop!)
    r = init_uniform(kg.nodes)
    for _ in range(50):                                   # iterate to convergence
        r = (1 - alpha) * kg.transition @ r + alpha * normalize(seeds)

    # 5. Rank passage nodes by their settled PageRank score
    return sorted(passage_nodes(r), key=lambda p: r[p], reverse=True)[:k]

Built on Prior Work

Results & Evidence

Setup: Llama-3.3-70B-Instruct as the QA reader and the triple extractor/filter; NV-Embed-v2 as the retriever. Seven datasets across three categories. Metric is F1 (answer quality) and recall@5 (retrieval quality).

Headline (F1, averaged across all 7 datasets):

• HippoRAG 2: 59.8 — best overall.
• NV-Embed-v2 (best plain embedding model): 57.0.
• Original HippoRAG: 53.1. RAPTOR: 54.9. GraphRAG: 56.1. LightRAG: 6.6 (collapses on simple/multi-hop QA).

The point it’s making: every structure-augmented predecessor underperformed the best plain embedding model overall. HippoRAG 2 is the first to beat it. The 7-point associativity claim is the gain on multi-hop tasks specifically. Crucially, it shows no regression on simple factual QA (NQ 63.3 vs NV-Embed 61.9; PopQA 56.2 vs 55.7) — that’s the whole thesis, validated.

Ablations (multi-hop recall@5, avg) — these are the most useful numbers for builders:

• Full HippoRAG 2: 87.1
• Query-to-node instead of query-to-triple: 59.6 (−27.5, the linking method is the single biggest lever)
• NER-to-node (v1 style): 74.6 (−12.5)
• Remove passage nodes: 81.0 (−6.1)
• Remove the LLM filter: 86.4 (−0.7, smallest contributor)
• Weight factor 0.05 is near-optimal; performance is fairly flat from 0.01–0.1, so it’s not knife-edge sensitive.

Caveats — read these before you sell it:

• The LLM triple filter contributes only 0.7 points of recall while adding an LLM call per query. The cost/benefit is thin; you may skip it in production.
• Every benchmark is Wikipedia-style QA. No enterprise data, no genuinely long conversations, no real-time updating tested at scale. “Continual learning” is claimed via the architecture (synonym edges merge new docs) but not stress-tested with a long stream of evolving documents.
• Indexing cost is real: an LLM OpenIE call per passage to build the graph. For a million-doc corpus that’s a meaningful one-time bill, and re-indexing churned docs adds up.
• Results lean on a 70B model for extraction; the paper notes robustness to other LLMs, but quality of the extracted graph gates everything downstream.

How You’d Use It

This is a drop-in upgrade for the retrieval layer of any RAG product where questions span multiple documents — which describes most real client knowledge bases.

• Multi-document Q&A for clients. Legal discovery, due-diligence over a data room, “what do all our incident reports say about root cause X” — these are exactly the multi-hop and sense-making cases where plain vector RAG quietly returns half the answer. HippoRAG 2’s PPR diffusion finds the connected passages.
• A premium “knowledge memory” tier. You can frame this as a capability, not a feature: “vector RAG answers what’s in one document; our graph memory answers what’s implied across all of them.” That’s a real, demonstrable differentiator in a sales demo — show a multi-hop question both systems get, and only yours connects.
• Agentic / MAS retrieval tool. In a multi-agent system, expose HippoRAG 2 as the shared long-term memory tool. Agents write findings as passages; the synonym-edge mechanism merges new findings with old; later agents query the same graph and get associative recall across the whole run. This is a cleaner shared-memory substrate than a flat vector store because it models relationships between what agents have learned.
• Where it slots in: replace your retriever, keep your reader LLM and your chunking. The graph build is an offline preprocessing job; query latency adds PPR (cheap, it’s sparse matrix iteration) plus optionally one LLM filter call.

Build Your Own (Minimal Recipe)

You can capture ~80% of the value with a weekend prototype. Skip the LLM filter first (it’s worth 0.7 points) and nail the two things that actually move the needle: passage-in-graph and query-to-triple.

Components & build order:

1. OpenIE extraction. For each passage, prompt any capable LLM: “extract (subject, relation, object) triples.” Store triples + a back-reference to the source passage.
2. Build the graph. Use networkx. Add phrase nodes (from subjects/objects), relation edges, passage nodes, and “contains” edges from each passage to its phrases.
3. Synonym edges. Embed phrases (any sentence-transformer or NV-Embed), add edges between pairs above a cosine threshold (start at 0.8).
4. Query-to-triple linking. Embed each triple as a sentence (“subject relation object”). At query time, embed the query and retrieve top-k triples by cosine — this is the high-value step, do not shortcut to NER.
5. Seed + PPR. Seed the personalization vector from the matched triples’ phrase nodes (full weight) plus passage nodes (weight ≈ 0.05). Run networkx.pagerank(G, personalization=seeds). Rank passage nodes by score.
6. Read. Feed top passages to your reader LLM.

The two genuinely hard parts:

• OpenIE quality. Garbage triples → garbage graph. Constrain the prompt, give few-shot examples, and validate on a sample. This is where most of your tuning time goes.
• Tuning the personalization weights. Getting the phrase-vs-passage balance right (the 0.05 factor) and the teleport α matters; build a small validation set of multi-hop questions and sweep.

Reach for: networkx (graph + PPR built in), a strong embedding model (NV-Embed-v2, GritLM-7B, or any BEIR-leaderboard model), an OpenIE-capable LLM (Llama-3.3-70B or any frontier model). The official code is at github.com/OSU-NLP-Group/HippoRAG.

How to Improve It

• Make the filter earn its keep. The LLM triple filter adds latency for 0.7 points. Replace it with a cheap cross-encoder reranker over triples, or a small fine-tuned classifier — same noise reduction, fraction of the cost. Test whether that recovers the gain without the per-query LLM call.
• Incremental / streaming indexing. The paper indexes a fixed corpus. True continual learning needs efficient incremental graph updates as documents arrive and change — including edge re-computation and stale-node eviction. This is the obvious productization gap and a real engineering moat if you solve it.
• Episodic memory for long conversations. The authors flag this as future work. Add temporally-ordered passage nodes and time-decay on edges so the graph models when things were said, not just what — turning it into agent conversation memory.
• Learned seed weighting. The phrase/passage weight is a single global constant (0.05). Learn it per-query (e.g., a tiny model that predicts whether a query is factual vs. multi-hop and shifts the weight accordingly) — factual queries want passage-dominant seeding, multi-hop wants phrase-dominant.
• Confidence-gated graph search. Run cheap dense retrieval first; only invoke OpenIE-graph + PPR when the top passages disagree or coverage is low. Most production queries are simple factual lookups — don’t pay the graph tax on them.

Glossary

• RAG (Retrieval-Augmented Generation) — give an LLM new knowledge by retrieving relevant text at query time and putting it in the prompt, instead of retraining.
• Non-parametric continual learning — learning new knowledge by changing an external store (the graph/corpus), never the model’s weights (parameters) — so no catastrophic forgetting.
• Catastrophic forgetting — when fine-tuning on new data erases previously learned knowledge.
• Associativity — the ability to connect facts spread across different documents (multi-hop reasoning).
• Sense-making — understanding a large, complex body of text where the answer is a synthesis, not a snippet.
• OpenIE (Open Information Extraction) — extracting (subject, relation, object) triples from text with no fixed schema; here, done by an LLM.
• Triple — a single fact as (subject, relation, object), e.g. (Spielberg, directed, Jaws); the atomic unit of the knowledge graph.
• Phrase node / passage node — graph nodes for extracted concepts vs. for whole original passages; HippoRAG 2’s key addition is putting both in one graph.
• Personalized PageRank (PPR) — PageRank biased to “teleport” back to chosen seed nodes, so importance scores reflect proximity to the query; the engine for multi-hop retrieval.
• Personalization / seed vector — the set of query-relevant entry nodes (and their weights) that PPR diffuses relevance out from.
• Recognition memory — here, an LLM step that judges which retrieved triples are actually relevant (recognition) vs. blindly using all of them (recall).
• Dense vs. sparse coding — neuroscience terms the authors borrow: dense = rich distributed context (passages), sparse = compact concepts (phrases); the graph fuses both.
• recall@5 / F1 — retrieval metric (is the right passage in the top 5?) vs. answer-quality metric (token overlap with the gold answer).