Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects

TL;DR

Most “agent memory” today is a thin layer bolted onto a stateless LLM: extract snippets from chat, dump them in a vector DB, retrieve top-k, paste into the prompt. That works for personalization but falls apart over long horizons — it blurs what the agent knows versus what it believes, can’t reason about when things happened, and gives no audit trail for why it answered a certain way. Hindsight reorganizes memory into four logical networks — world facts, the agent’s own experiences, subjective opinions (with confidence scores), and synthesized entity summaries — and exposes three operations: retain (turn transcripts into a temporal, entity-linked graph), recall (four-way parallel retrieval with rank fusion and reranking under a token budget), and reflect (preference-conditioned answering that forms and updates opinions). On LongMemEval, this takes a plain 20B open model from 39.0% → 83.6%, surpassing full-context GPT-4o (60.2%); with a bigger backbone it hits 91.4%, the best across all systems tested. The headline finding: the memory architecture, not raw model size, carries the performance.

Problem & Motivation

You’re building an agent meant to act like a long-term partner — remember last month’s conversation, track facts about the world, hold a stable point of view. The standard recipe is RAG-over-chat: an extractor pulls “salient snippets” from conversations, embeds them, drops them in a vector or graph store, and at query time retrieves the top-k most similar chunks into the prompt of an otherwise stateless model.

This breaks in three concrete, recurring ways the paper names directly:

1. It can’t preserve and granularly access long-term info across sessions. Over 50–500 sessions and a million-plus tokens, pure semantic top-k drowns the relevant fact in near-duplicates. Multi-session questions (“what did I decide across these three chats?”) and temporal questions (“what did I say before the trip?”) are exactly where vector search is weakest, because the answer isn’t the most semantically similar chunk — it’s the one connected by time or shared entity.
2. It can’t tell observation from belief. A flat store treats “Alice is a software engineer at Google” (a fact) and “Python is the best language for data science” (an opinion) identically. The agent has no structural way to say “this I observed, that I merely concluded” — which is fatal for traceability and for anything an auditor or client needs to trust.
3. It can’t hold a stable viewpoint. Without an explicit place to store beliefs and update them, the agent produces locally plausible but globally inconsistent answers — confident “remote work is great” one session, “remote work is risky” the next, with no memory of having formed either view.

Prior systems each fix a slice: MemGPT pages text in/out like an OS but keeps memory as unstructured blocks; Zep builds temporal knowledge graphs but only of objective facts; A-Mem makes evolving atomic notes but treats all memory uniformly; Mem0 optimizes production retrieval but resolves conflicts by overwriting rows, not evolving beliefs. None of them separate evidence from inference and model evolving opinions and stay fully external (no fine-tuning). That gap is the paper’s target.

What’s New (Core Contribution)

Four genuine contributions. The novelty is mostly organizational — this is a systems/architecture paper, not a new model — but the organization is the point.

• A four-network memory partition with distinct epistemic roles. Before: one undifferentiated pile of “memories.” Now: memory is split into World (objective external facts), Experience (the agent’s own first-person actions/recommendations), Opinion (subjective judgments, each with a confidence score and timestamp), and Observation (preference-neutral entity summaries synthesized from facts). This is the spine — it lets the system structurally separate “what I saw” from “what I believe.”
• Three first-class operations: retain / recall / reflect. Before: memory is “write to store” + “top-k read.” Now: Retain(B,D) ingests data into the typed graph and reinforces existing beliefs; Recall(B,Q,k) is a multi-strategy retrieval under a token budget; Reflect(B,Q,Θ) generates a preference-shaped answer and forms/updates opinions. Memory becomes a substrate with verbs, not a passive index.
• TEMPR: temporal, entity-aware retrieval that goes beyond top-k. Before: fixed top-k semantic search. Now: four channels run in parallel — semantic (vectors), keyword (BM25), graph (spreading activation over entity/causal/temporal/semantic edges), and temporal (date-range matching) — fused with Reciprocal Rank Fusion, reranked by a cross-encoder, and packed to fit a caller-specified token budget. Narrative (“coarse”) fact extraction keeps each memory self-contained instead of fragmenting an exchange into five brittle snippets.
• CARA: preference-conditioned reasoning with an evolving opinion network. Before: personality is a one-off prompt string. Now: each memory bank carries a disposition profile Θ = (Skepticism, Literalism, Empathy each 1–5, plus a bias-strength β∈[0,1]). The same facts produce systematically different opinions depending on Θ, and opinions are reinforced or weakened over time via a confidence-update rule rather than overwritten.

Be honest about the hype: the individual ingredients (BM25+vector hybrid, RRF, cross-encoder reranking, knowledge-graph memory, confidence-scored beliefs) all exist in prior work. What’s new is combining them under a clean epistemic abstraction and showing it moves the needle a lot on long-horizon benchmarks with a small model.

How It Works (Technically)

Think of Hindsight as two subsystems sitting on one typed graph. TEMPR owns retain + recall (building and querying memory). CARA owns reflect (reasoning over it with a personality and updating beliefs). Let’s trace one conversation all the way through.

The memory unit and the four networks

Every fact is stored as one self-contained node:

f = (u, b, t, v, τs, τe, τm, ℓ, c, x)

In plain English: u = unique id, b = which memory bank it belongs to, t = the narrative text (a full sentence or two, not a fragment), v = its embedding vector, (τs, τe) = the time interval the event occurred, τm = when it was mentioned, ℓ = which of the four networks it lives in, c = an optional confidence (only opinions use it), x = extras (access counts, full-text search vectors). The split between occurrence time (τs, τe) and mention time τm is what enables real temporal reasoning — “what happened in June” is a query over occurrence intervals, independent of when it was discussed.

The four networks M = {W, B, O, S} just partition where a fact lands, by its epistemic role:

• W (World): “Alice works at Google.”
• B (Experience): “I recommended Yosemite to Alice.” (first person, the agent’s own history)
• O (Opinion): ("Python is best for data science", c=0.85, τ) — a tuple with belief strength.
• S (Observation): “Alice is an ML software engineer at Google” — a synthesized profile, regenerated in the background when underlying facts change.

Retain: turning a transcript into a graph

The key design choice is narrative (coarse-grained) extraction: an LLM reads a whole exchange and emits 2–5 self-contained facts, each preserving who-said-what and why, rather than one fact per sentence. The paper’s example: instead of five brittle fragments (“Bob suggested Summer Vibes”, “Alice wanted something unique”, …), it stores one narrative fact that captures the entire playlist-naming discussion and its conclusion. This makes retrieval robust to where you happened to split the text.

Internally retain does six things: coreference resolution, temporal normalization (“last week” → absolute (τs, τe)), participant attribution, preserving stated reasoning, classifying the fact into one of the four networks, and entity extraction (PERSON/ORG/LOCATION/PRODUCT/CONCEPT/OTHER).

Then it builds the graph G=(V,E). Entity resolution maps each mention m to a canonical entity by maximizing a weighted similarity (Eq. 2):

ρ(m) = argmax_e [ α·sim_str(m,e) + β·sim_co(m,e) + γ·sim_temp(m,e) ]

That’s just: pick the canonical entity that best matches on string similarity (Levenshtein), co-occurrence with the same neighbors, and temporal proximity, with tunable weights α,β,γ. Once entities are canonical, four edge types get laid down:

• Entity links (weight 1.0): any two facts mentioning the same entity get a bidirectional edge. This is what lets you connect “Alice” conversations across months that share no words.
• Temporal links: weight decays with time gap, w = exp(−Δt/σt) — closer-in-time facts are more strongly linked.
• Semantic links: cosine similarity above a threshold θs.
• Causal links (weight 1.0): cause→effect relations the LLM flags (causes, enables, prevents, …), deliberately upweighted during traversal so explanations surface.

Observations (the S network) are generated asynchronously: o_e = SummarizeLLM(F_e) over all facts mentioning entity e. Writes stay fast; entity profiles improve in the background.

Recall: four-way retrieval under a token budget

Unlike a fixed top-k API, recall takes a token budget k (and optional latency budget) so a caller can ask for “just enough” memory. Four channels run in parallel:

1. Semantic — cosine similarity over an HNSW/pgvector index. Catches paraphrases.
2. Keyword (BM25) — full-text over a GIN index. Catches exact proper nouns/IDs the embedding blurs.
3. Graph (spreading activation) — start from top semantic hits, then propagate activation along edges (Eq. 12): A(fj, t+1) = max over edges [ A(fi,t) · w · δ · μ(ℓ) ]. Read this as: a node lights up from its strongest neighbor, attenuated by edge weight w, a decay δ, and a link-type multiplier μ(ℓ) — causal/entity edges get μ>1 (boosted), weak/long-range edges μ≤1. This surfaces facts that aren’t textually similar to the query but are connected.
4. Temporal — if the query has a date constraint, a rule-based parser (with a flan-t5-small fallback) turns “last weekend” into [τstart, τend], and only facts whose occurrence interval overlaps are kept, scored by midpoint proximity (Eq. 14).

The four ranked lists are merged with Reciprocal Rank Fusion (Eq. 15): RRF(f) = Σ_i 1/(k + r_i(f)) where r_i(f) is f’s rank in list i (∞ if absent), and k≈60. The intuition: a memory ranked near the top of several channels accumulates the most score; RRF is rank-based so it doesn’t need the four channels’ raw scores to be calibrated against each other, and a missing item simply contributes zero rather than being penalized. Then a cross-encoder reranker (ms-marco-MiniLM-L-6-v2) jointly encodes query+candidate for precision, and finally a greedy token-budget packing step adds facts in rank order until the next one would blow the budget k.

Reflect: personality-shaped answering + belief updates

CARA loads the bank’s profile Θ=(S,L,E,β), verbalizes the numeric dispositions into a system-prompt sentence (ϕ(Θ) → “You are generally trusting, interpret language flexibly, and are highly empathetic…”), calls recall to fetch F_Q, and generates a response whose tone/reasoning are shaped by Θ — with β dialing how hard to lean in (β=0 → fact-first/objective, β=1 → strongly opinionated). The paper’s clean demonstration: identical facts about remote work yield “net positive, removes commute, enables flexibility” under (S=1,L=2,E=5) versus “risks undermining performance, harder to maintain oversight” under (S=5,L=5,E=1).

During reflection the agent may form new opinions o=(t,c,τ,b,E). And when new facts arrive later, opinion reinforcement updates existing beliefs in three steps: (1) find candidate opinions sharing an entity or above an embedding-similarity threshold; (2) have the LLM Assess(o,f) classify the new evidence as reinforce / weaken / contradict / neutral; (3) nudge the confidence with a step size α (Eq. 26):

reinforce  → c' = min(c + α, 1)
weaken     → c' = max(c − α, 0)
contradict → c' = max(c − 2α, 0)   # and optionally rewrite the opinion text
neutral    → c' = c

Small α keeps beliefs from oscillating on a single example while letting repeated evidence move them — opinions become trajectories, e.g. “Python is best for DS” (0.70 → 0.85 as AI/ML ecosystem evidence arrives → 0.55 “strong but has trade-offs” as Julia/Rust evidence accumulates). A parallel background merging step (h' = MergeLLM(h, h_new)) keeps the agent’s first-person bio coherent — resolving “born in Texas” vs. “born in Colorado” in favor of the new info, appending non-conflicting detail, staying concise.

Architecture & data flow

flowchart TB
  D[Conversation transcript D] --> EX[LLM narrative fact extraction<br/>2-5 self-contained facts]
  EX --> CL[Classify into network<br/>World / Experience / Opinion / Observation]
  CL --> ER[Entity resolution + link construction<br/>entity / temporal / semantic / causal edges]
  ER --> BANK[(Memory bank B: graph G=V,E<br/>+ profile Theta)]
  BANK -. background .-> OBS[Observation synthesis<br/>SummarizeLLM per entity]
  OBS --> BANK

  Q[Query Q + token budget k] --> R4{Four-way parallel recall}
  BANK --> R4
  R4 --> SEM[Semantic / vectors]
  R4 --> BM[BM25 keyword]
  R4 --> GR[Graph spreading activation]
  R4 --> TP[Temporal range match]
  SEM --> RRF[Reciprocal Rank Fusion]
  BM --> RRF
  GR --> RRF
  TP --> RRF
  RRF --> CE[Cross-encoder rerank]
  CE --> PACK[Token-budget packing <= k]
  PACK --> REF[CARA reflect:<br/>verbalize Theta + facts -> response r]
  REF --> RESP[Answer r]
  REF --> OPN[Form / reinforce opinions -> O']
  OPN --> BANK

Schematic of the four-network split. Click a fact to see which network it lands in and why — facts vs. experiences vs. confidence-scored opinions vs. synthesized observations.

Spreading activation over the memory graph. The query lights up its best semantic match, then activation propagates along entity/causal/temporal edges (boosted or decayed by the link-type multiplier μ), surfacing connected facts that pure top-k vector search would miss.

The algorithm, simplified

# Hindsight's core loop: retain -> recall -> reflect over a typed memory graph.
# llm(prompt)->str, embed(x)->vec, judge(o,f)->str are stubbed model calls.

def retain(bank, transcript):
    facts = llm(EXTRACT_PROMPT, transcript)        # 2-5 NARRATIVE facts, not fragments
    for f in facts:
        f.network = classify(f)                    # World / Experience / Opinion / Observation
        f.entities = resolve(f.mentions, bank)     # map mentions -> canonical entities (Eq.2)
        f.vec = embed(f.text)
        bank.add(f)
        link(bank, f)                              # entity / temporal / semantic / causal edges
        reinforce_opinions(bank, f)                # beliefs evolve as evidence arrives (Eq.26)
    schedule_observation_refresh(bank, facts)      # async entity summaries

def recall(bank, query, k):                        # k = TOKEN budget, not fixed top-k
    qv = embed(query)
    cands = parallel(                              # four notions of relevance, in parallel
        semantic(bank, qv), bm25(bank, query),
        graph_spread(bank, qv), temporal(bank, query))
    fused = rrf(cands, c=60)                        # 1/(c+rank) summed across lists (Eq.15)
    ranked = cross_encoder_rerank(query, fused)    # joint query-doc scoring for precision
    out, used = [], 0
    for f in ranked:                               # greedy pack until budget hit (Eq.17)
        if used + tokens(f) > k: break
        out.append(f); used += tokens(f)
    return out

def reflect(bank, query, theta):                   # theta = (skepticism, literalism, empathy, beta)
    facts = recall(bank, query, k=BUDGET)
    sys = verbalize(bank.name, bank.background, theta)  # numeric disposition -> prompt sentence
    r, new_ops = llm(sys, facts, beta=theta.beta)  # response + candidate opinion updates
    for o in new_ops: bank.opinions.add(o)         # opinions stored as (text, confidence, time)
    return r

def reinforce_opinions(bank, f):
    for o in bank.opinions.related(f):             # share an entity or high embedding sim (Eq.25)
        rel = judge(o, f)                          # reinforce / weaken / contradict / neutral
        o.confidence = step(o.confidence, rel)     # +a / -a / -2a / 0, clamped to [0,1]

Built on Prior Work

Results & Evidence

Benchmarks. LongMemEval (500 questions; S setting ≈115k tokens/50 sessions, M setting ≈1.5M tokens/500 sessions) tests information extraction, multi-session reasoning, temporal reasoning, knowledge update, and abstention. LoCoMo (50 human-human conversations, ~305 turns / ~19 sessions each, includes images) tests recall of personal details and events across distant sessions. Evaluation is LLM-as-a-judge (GPT-OSS-120B, temp 0), binary correct/incorrect.

Headline numbers.

• LongMemEval: Hindsight with OSS-20B (same model used everywhere — extraction, reflection) hits 83.6% overall, a +44.6 point jump over the full-context OSS-20B baseline (39.0%), and beats full-context GPT-4o (60.2%). With OSS-120B it reaches 89.0%, and with Gemini-3 as the answer generator, 91.4% — best across all systems, beating Supermemory+GPT-5 (84.6%). The biggest gains are exactly where the design targets: multi-session 21.1%→79.7%, temporal 31.6%→79.7%, preference 20.0%→66.7%.
• LoCoMo: Overall rises from Memobase’s 75.78% to 83.18% (OSS-20B), 85.67% (OSS-120B), 89.61% (Gemini-3) — the last effectively matching Backboard’s claimed 90.00% while using a fully open-source memory stack, and taking the top Open-Domain score (95.12%).

What the evidence does establish: the architecture, not model scale, is doing the work — the OSS-20B base goes from 39% to 83.6% with only the memory layer added, on the same backbone. That’s a clean ablation and the paper’s strongest claim.

Caveats — read these before you quote the numbers:

• Baselines are mostly reported, not reproduced. LongMemEval baselines come straight from Supermemory’s technical report (using their GPT-4o judge); LoCoMo baselines from Backboard’s published figures, explicitly “could not be independently reproduced.” So the comparison mixes judge models and harnesses — Hindsight’s own runs use a GPT-OSS-120B judge.
• Token budgets are redacted in the extracted text (” tokens”), so retrieval coverage isn’t fully specified in this version.
• Experiments use neutral profiles (S=L=E=3, β=0.2) — the CARA preference/opinion machinery, a major selling point, is essentially turned off for the benchmarks, which only test factual recall. There’s no quantitative evaluation of opinion consistency, belief evolution, or disposition control — the most novel parts are shown only via hand-picked qualitative examples.
• No latency/cost numbers for the four-way retrieval + reranking + async observation pipeline, which matters a lot for production.

How You’d Use It

For an AI services company this is a blueprint for a stateful agent memory tier you can sell as a capability, and a useful counter-narrative to “just use a bigger model.”

• Long-lived assistants / copilots. Any client agent that spans sessions (support, sales, internal knowledge worker) benefits from the temporal + entity-graph recall. The multi-session and temporal gains are the ones clients actually feel (“it forgot what we agreed last week”).
• The epistemic split is a compliance/trust feature you can charge for. Keeping World/Experience (evidence) structurally separate from Opinion (inference), with confidence scores and timestamps, gives you an audit trail — “the agent answered X because of these observed facts, and this is a belief it holds at confidence 0.6 formed on date Y.” That’s exactly what regulated clients ask for and most vector-store memory can’t provide.
• CARA = configurable agent persona, done properly. Instead of brittle “act friendly” prompt strings, you expose three sliders (skepticism/literalism/empathy) + a bias dial per client or per agent role. A skeptical, literal “reviewer” agent and an empathetic, flexible “concierge” agent share the same facts but reason differently — a clean product surface.
• Small-model deployability is the commercial punchline. 83.6% on a 20B model that “runs on a single high-end consumer GPU” means you can offer strong long-horizon memory without per-query frontier API cost — a real margin and data-sovereignty story.
• Where it slots in: drop TEMPR between your transcript ingestion and your LLM call (it’s an external memory service; no fine-tuning), and CARA wraps your generation step.

Build Your Own (Minimal Recipe)

You can capture ~80% of the value without the full graph machinery. Build in this order:

1. Typed extraction. One LLM call per session with a structured-output schema that emits 2–5 narrative facts, each tagged network ∈ {world, experience, opinion, observation}, normalized (τs, τe), and an entity list. This single change — coarse narrative facts + a type tag — gives you the epistemic split and most of the robustness. (Library: any LLM with JSON/structured output.)
2. Hybrid recall = semantic + BM25 + RRF. Postgres with pgvector (HNSW) for vectors and a GIN full-text index for BM25; fuse with the 5-line RRF formula. This alone beats top-k vector search. Add a cross-encoder reranker (ms-marco-MiniLM-L-6-v2, runs on CPU/cheap GPU) for precision.
3. Token-budget packing instead of fixed top-k — trivial greedy loop, immediately useful.
4. Then add the graph: entity links (resolve mentions to canonical entities), temporal links, causal links, and spreading activation. This is the part that lifts multi-session/temporal scores and is the first genuinely hard part — entity resolution quality gates everything downstream.
5. CARA last: opinion network with confidence + the reinforce/weaken/contradict update rule, plus verbalized disposition prompts. The second hard part is making opinion reinforcement stable (tuning α and the Assess classifier so beliefs neither freeze nor oscillate).

Models to reach for: GPT-OSS-20B (or any solid 14–32B open model) for extraction+reflection; a small embedding model; flan-t5-small only if you need the temporal-parse fallback. Skip the async observation synthesis until you have entity-heavy “tell me about X” traffic.

How to Improve It

1. Actually evaluate the novel half. The opinion/disposition machinery is untested quantitatively. Build a benchmark for preference consistency (does the agent hold a stable, configured viewpoint across N sessions?) and belief calibration (do confidence scores track ground-truth correctness?). Right now those claims rest on two cherry-picked figures.
2. Close the RL loop the conclusion hand-waves at. Extraction, graph construction, and retrieval are fixed pipelines tuned by hand. Treat recall as a policy and use the downstream QA-judge signal as reward (à la Memory-R1) to jointly learn what to extract and what to retrieve — likely the biggest robustness win in noisy open-domain data.
3. Controlled forgetting + privacy. A long-lived bank grows unbounded and accumulates stale/sensitive facts. Add time-aware decay, a “retract this entity” operation, and belief-revision that can lower confidence on contradiction without losing the trajectory — the paper lists this as future work; it’s table-stakes for production.
4. Adaptive budget per query. The token budget k is currently a static config. Pair recall with a query-difficulty estimator (a cheap classifier or even the bandit idea from routing work) so simple questions spend little and multi-hop questions spend more — directly attacks the latency/cost gap the paper never measures.
5. Make the cross-encoder temporal-native and the graph multiplier learned. μ(ℓ) (the link-type boosts) and the RRF constant are hand-set; learn them from feedback. And the reranker currently gets temporal info only as appended text — a reranker that natively scores temporal overlap would sharpen the TR category further.

Glossary

• Agent memory — an external store + retrieval layer that lets a stateless LLM “remember” across turns/sessions.
• RAG (retrieval-augmented generation) — fetch relevant snippets and paste them into the prompt before generating.
• Top-k retrieval — return the k most similar items; the conventional, fixed-size memory read this paper moves beyond.
• Four-network organization — Hindsight’s split of memory into World (facts), Experience (agent’s own history), Opinion (beliefs w/ confidence), Observation (entity summaries).
• TEMPR — the subsystem implementing retain + recall: builds the temporal entity graph and does multi-strategy retrieval.
• CARA — the subsystem implementing reflect: preference-conditioned generation + opinion formation/reinforcement.
• Narrative (coarse) fact extraction — emitting 2–5 self-contained, context-preserving facts per exchange instead of one fragment per sentence.
• Entity resolution — mapping different surface mentions (“Alice”, “she”, “Ms. Smith”) to one canonical entity so facts can be linked.
• Spreading activation — graph search that propagates a relevance “charge” from query-matched nodes along edges to surface connected facts.
• Link-type multiplier μ(ℓ) — per-edge-type boost/decay (causal & entity edges boosted, weak/long-range decayed) used during graph traversal.
• BM25 — a classic keyword/lexical ranking function; strong on exact proper nouns the embedding blurs.
• Reciprocal Rank Fusion (RRF) — combine multiple ranked lists by summing 1/(c+rank); rank-based so it needs no score calibration.
• Cross-encoder reranker — a model that jointly encodes (query, candidate) for a precise relevance score, slower but sharper than vector similarity.
• Disposition profile Θ=(S,L,E,β) — Skepticism, Literalism, Empathy (1–5) plus bias-strength β (0–1) that shapes how CARA reasons and how strongly.
• Opinion reinforcement — the rule that nudges a belief’s confidence up/down (±α, −2α on contradiction) as new evidence is assessed.
• Occurrence vs. mention time — when an event happened (τs,τe) vs. when it was talked about τm; separating them enables real temporal queries.
• LLM-as-a-judge — using a separate LLM to score whether a generated answer matches ground truth; the evaluation method here.
• LongMemEval / LoCoMo — the two long-horizon conversational-memory benchmarks used for evaluation.