Memento: Fine-tuning LLM Agents without Fine-tuning LLMs

TL;DR

Most ways to make an LLM agent “learn” either hardcode a fixed workflow (rigid, never improves) or fine-tune the model with RL/SFT (expensive, slow, risks forgetting). Memento takes a third path: freeze the LLM entirely and bolt on an external memory of past episodes (a “Case Bank”). When a new task arrives, the agent retrieves similar past cases — including failures — and conditions its plan on them. The genuinely new part is that the retrieval itself is learned: a small neural network (a Q-function, trained by a one-step reinforcement-learning rule) scores which stored cases are actually useful, rather than just grabbing the most textually similar ones. With GPT-4.1 planning and o3/o4-mini executing, Memento tops the GAIA validation leaderboard (87.9% Pass@3), hits 95% on SimpleQA, and adds 4.7–9.6 points on out-of-distribution tasks purely from accumulated memory — no gradient updates to the LLM.

Problem & Motivation

Here is the concrete pain. You deploy an agent for a client. It does deep research — searches the web, reads PDFs, runs code, chains 5 to 50 tool calls per task. Today you have two unappealing options for making it improve over time:

1. Hardcode the workflow. Fast to ship, but the agent is frozen the moment it deploys. It can’t absorb new information or adapt to a task shape it hasn’t seen. Every edge case is a code change.
2. Fine-tune the underlying LLM (supervised fine-tuning or RL like GRPO). This does let behavior adapt, but it is brutal in practice: you must roll out thousands of trajectories to gather training data, you need large volumes of human-annotated questions, each update costs serious compute, and you risk catastrophic forgetting (teaching it the new task degrades old skills). For an open-ended agent that should learn continuously, retraining the weights after every batch of experience is a non-starter.

The research question the paper poses verbatim: How can we build LLM agents that learn continuously from a changing environment without the prohibitive cost of fine-tuning the underlying LLMs?

The motivation is borrowed from human cognition: we improve not by rewiring our neurons after every task but by remembering specific past episodes and reasoning by analogy (“this is like that customer issue from last month”). That is case-based reasoning (CBR) — a decades-old AI idea the paper resurrects and makes learnable.

What’s New (Core Contribution)

1. A formalism: the Memory-Based MDP (M-MDP). Before: agent memory was an ad-hoc bag of retrieved snippets glued onto a prompt. Now: memory is a first-class element of the decision process. They define an MDP tuple with an explicit memory space alongside states, actions, rewards. The agent’s policy is split into two factors: a retrieval policy (which past case to pull) and the frozen LLM (which action to take given that case). This cleanly separates “what to remember” from “what to do.”
2. Learned, not just similarity-based, retrieval. Before: RAG and prior CBR agents retrieve by raw text similarity — nearest neighbors in embedding space. Now: Memento learns a Q-function that estimates how useful recalling a given case will be (probability it leads to a correct answer), via online soft Q-learning. The optimal retrieval policy is a softmax over these Q-values. This is the “fine-tuning without fine-tuning the LLM” — you train a tiny external model, not the 100B-parameter backbone.
3. Two concrete memory variants with a clean degradation. A non-parametric version (append-and-cosine-retrieve, zero training) and a parametric version (learns the Q-function as a 2-layer MLP). They show the multi-step Q-learning collapses to a simple binary classification loss in the single-step planning case — making it cheap and stable to train.
4. A working deep-research system + SOTA results. A planner–executor architecture using the Model Context Protocol (MCP) for tools, hitting top-1 on GAIA validation and beating the best training-based web agents on DeepResearcher — while learning purely through memory.

The honest read: CBR, episodic memory, and planner–executor agents all pre-existed. The real novelty is casting case retrieval as a learnable RL policy with a closed-form softmax solution and showing it beats naive similarity retrieval. The rest is competent systems engineering.

How It Works (Technically)

Memento runs a loop with five stages per step. Walk one task through it: “In a YouTube video posted in June, what is the character’s first name?”

1. Retrieve. The planner encodes the current task into a vector, queries the Case Bank, and pulls the top-K most useful past cases (each case = (state, plan, success-flag)). Maybe it recalls a past case where “find info from a YouTube video” was solved by getting the video ID then examining the thumbnail.
2. Reuse & Revise. The retrieved cases are concatenated into the prompt. The frozen LLM (GPT-4.1) generates a plan: decompose into subtasks — identify the YouTube ID, examine the thumbnail, read the first name.
3. Execute. A separate executor LLM (o3/o4-mini) takes each subtask and acts as an MCP client, calling tools (search, crawler, video summarizer, code sandbox) until it returns a result. The planner replans if the task isn’t done.
4. Evaluate & Retain. On task completion, the outcome (success or failure) is written back as a new case: Write(s, a, r) → Case Bank grows. Crucially, failures are stored too — so the agent learns what not to do.
5. Transition. State advances; the loop repeats. Over many tasks the Case Bank becomes a transferable repository of experience.

The two-LLM split matters: the planner is the only CBR agent (it does the learning-by-memory), the executor is a stateless tool-caller. This keeps the learnable surface small.

Demystifying the math

The agent’s overall policy (Eq. 1) is:

π(a | s, M) = Σ_c μ(c | s, M) · p_LLM(a | s, c)

Plain English: the probability of taking action a = sum over every stored case c of [probability you retrieve case c] × [probability the LLM produces a after seeing case c]. The LLM part p_LLM is frozen. The only thing we get to optimize is μ, the retrieval policy — which case to surface. That’s the whole game.

To learn μ, they use maximum-entropy RL (Eq. 3). The objective rewards getting the task right plus an entropy bonus α·H(μ):

maximize: expected reward + α · (entropy of the retrieval distribution)

Why the entropy term? Without it, the policy would collapse to always retrieving the single highest-scoring case, never exploring. The entropy bonus keeps retrieval diverse — it hedges across several plausibly-useful cases. α tunes how much you value exploration. (This is the same trick behind Soft Actor-Critic in robotics RL.)

The payoff is a closed-form optimal retrieval policy (Eq. 7):

μ*(c | s, M) = softmax over cases of [ Q*(s, M, c) / α ]

Plain English: retrieve case c with probability proportional to exp(its Q-value / α). So if you can learn the Q-function Q(s, M, c) — “how good is it to recall case c in state s” — retrieval is just a softmax. No search, no heuristics.

How do you learn Q? In general, temporal-difference (TD) learning (Eq. 8) — the standard RL update where you nudge Q toward reward + discounted future value. But natural-language states make raw TD hard, so they offer two simplifications:

• Kernel / episodic-control estimate (Eq. 9): estimate Q(s, ·, c) as a similarity-weighted average of recorded Q-values from past interactions that used the same case c. A small kernel network k_θ(s, s') learns the similarity. This is “non-parametric-ish” — Q is computed from stored experience, the only learned thing is the similarity metric.
• Single-step collapse (the parametric version, the one that works best): because CBR is applied only at planning time (one decision per task, not a long chain), the TD target collapses to just the immediate reward. There’s no bootstrapping, no moving target. Q-learning reduces to supervised learning. And since the reward is binary (correct = 1, wrong = 0), they swap mean-squared-error for cross-entropy loss (Eq. 15):

ℒ(θ) = −[ r·log Q(s,c;θ) + (1−r)·log(1 − Q(s,c;θ)) ]

Plain English: this is exactly logistic-regression / binary-classifier training. Q(s,c;θ) is interpreted as p(this case leads to success | state s, case c). You’re training a 2-layer MLP to predict “is recalling this case going to help?” That is the entire learnable component — a tiny classifier sitting on top of frozen sentence embeddings (SimCSE). Read then just takes the top-K cases by predicted Q-value (Eq. 16).

So “fine-tuning without fine-tuning the LLM” is literal: you train a small binary classifier over (state, case) pairs to gate retrieval. The LLM never sees a gradient.

Architecture & data flow

flowchart LR
  Q[User Query] --> P[Planner<br/>GPT-4.1 · CBR agent]
  CB[(Case Bank<br/>episodic memory)] -->|Read: top-K useful cases| P
  P -->|decomposed subtasks| SM[Subtask Memory]
  SM --> E[Executor<br/>o3 / o4-mini · MCP client]
  E -->|tool calls| T[MCP Tools<br/>search · crawl · code · video · image]
  T -->|results| E
  E -->|subtask results| P
  P -->|replan if unfinished| SM
  P -->|on completion: Write s,a,r| CB
  CB -->|online update| QF[Q-function<br/>2-layer MLP]
  QF -.->|scores cases for Read| CB
  P --> ANS[Final Answer]

Schematic of learned vs. similarity-based retrieval. Each dot is a stored case positioned by embedding similarity to the query (center). Color = the learned Q-value (usefulness). Toggle modes: similarity grabs the geometrically nearest cases; the learned policy grabs the highest-Q cases — which can be farther away but actually more useful. This is the paper's core mechanism in one picture.

The algorithm, simplified

# The Memento planning loop. The LLM is frozen; only `q_net` ever trains.
# case = (state_text, plan, reward) ; reward in {0,1}
def memento_step(task, case_bank, q_net, executor, K=4):
    s = encode(task)                                  # SimCSE sentence embedding

    # --- Read: learned retrieval (parametric variant) ---
    scored = [(c, q_net(s, encode(c.state))) for c in case_bank]   # p(useful | s, c)
    cases  = top_k(scored, K)                          # highest predicted usefulness
    # (non-parametric variant would instead do: top_k by cosine(s, encode(c.state)))

    # --- Reuse & Revise: frozen LLM plans, conditioned on recalled cases ---
    plan = llm_plan(task, retrieved=cases)             # GPT-4.1, no gradients ever
    subtasks = plan.subtasks

    # --- Execute: stateless tool-caller drives MCP tools to completion ---
    results = []
    for sub in subtasks:
        results.append(executor.run(sub))              # o3/o4-mini as MCP client
    answer, success = llm_aggregate(task, results)     # success: did we solve it?

    # --- Retain: store BOTH wins and losses; train the retrieval gate ---
    new_case = Case(state=task, plan=plan, reward=int(success))
    case_bank.append(new_case)
    # single-step Q-learning == binary classification on "was this case useful"
    loss = bce(q_net(s, encode(new_case.state)), target=new_case.reward)
    q_net.step(loss)                                   # the ONLY thing fine-tuned

    return answer

The thing to internalize: the “training” is one cheap gradient step on a small MLP per task. Everything expensive (the LLM) is read-only.

Built on Prior Work

Results & Evidence

What they tested. Four benchmarks chosen for different stresses: GAIA (long-horizon, up to ~50 steps, multi-tool), DeepResearcher (7 open-domain QA sets, live web), SimpleQA (4,330 single-hop factual, hallucination test), HLE / Humanity’s Last Exam (2,500 frontier expert questions). Planner = GPT-4.1; executor = o3 (GAIA) or o4-mini.

Headline numbers.

• GAIA: 87.88% Pass@3 → top-1 on validation among open-source frameworks; 79.40% on private test (4th). Beats Manus, AWorld, OWL, Alita.
• DeepResearcher: 66.6% F1 / 80.4% PM average — beats the best training-based agent (DeepResearcher itself) and nearly doubles CoT+RAG (37.7% F1). Strong claim: a memory agent with no weight updates outperforms an RL-fine-tuned one.
• SimpleQA: 95.0% — new SOTA over WebSailor (93.5%), WebDancer (90.5%).
• HLE: 24.4% PM — second to GPT-5 (25.3%), ahead of Gemini-2.5-Pro and o3-high.
• The CBR ablation (the load-bearing evidence): going Offline→Online executor→+Planning→+CBR, case-based memory adds a consistent +4.5/+7.0 (HLE), +3.7/+5.3 (SimpleQA), +6.7/+8.2 (DeepResearcher) F1/PM. OOD gains of 4.7–9.6 points purely from cases collected on different training datasets.

Caveats — read these before quoting the numbers to a client.

• The lift from CBR is real but modest. The big jumps come from planning + live tools (e.g., +29 F1 on DeepResearcher from adding planning). CBR adds single-digit points on top. The headline “beats training-based methods” is mostly the planner–executor+MCP scaffold; memory is the cherry, not the cake.
• It plateaus fast. Table 4: across 5 learning iterations, accuracy moves ~80.5%→85.4%, with most gain by iteration 3. With only ~3k tasks the Case Bank saturates — they explicitly say they couldn’t find meaningful gains after a few iterations. “Continual learning” here is more “warm-up then flat” than “unbounded improvement.”
• Small memory is better. K=4 is optimal; K=8/16/32 plateau or decline. More memory = more noise. So the win depends on good curation, which they don’t fully solve.
• Online tools can hurt. On DeepResearcher, adding web tools without planning dropped F1 by 18 points — they attribute this to data contamination and note internal model knowledge sometimes beats retrieval. Honest, but it means results are sensitive to benchmark hygiene.
• Backbone-dependent. Everything rides on frontier closed models (GPT-4.1, o3). HLE results show that without domain knowledge already in the backbone, neither tools nor memory rescue you. This is not a small-model technique.

What the evidence does establish: learned case retrieval reliably beats no-memory and similarity-only memory on these tasks, and you can get competitive deep-research performance with zero LLM fine-tuning. What it does not: that the approach scales to truly lifelong learning (the plateau), or that it works off frontier proprietary models.

How You’d Use It

This maps almost perfectly onto an AI-services agent offering. Three concrete slots:

1. A “learns-on-the-job” deep-research agent as a product. The pitch to clients writes itself: an agent that gets measurably better at their recurring research/analysis tasks over the first few weeks — without you retraining anything or shipping their data to a fine-tuning pipeline. The Case Bank is per-client, portable, inspectable (it’s just (task, plan, outcome) rows), and a clean differentiator vs. a generic ChatGPT wrapper.
2. Memory layer for an existing multi-agent system (your ARC MAS). You already have agent coordination working. Memento’s planner is a drop-in upgrade to whichever agent does task decomposition: give it a Case Bank scoped to that role, retrieve past decompositions, and gate retrieval with the tiny Q-classifier once you have a few hundred labeled outcomes. The non-parametric variant (cosine over embeddings) needs zero training and gives most of the benefit on day one.
3. MCP-native tool execution. The executor design — a stateless LLM acting as an MCP client over a registry of tools — is the cleanest current pattern for tool use and is exactly what the Anthropic ecosystem is standardizing on. Worth adopting independent of the memory idea.

The commercial moat is the accumulated, labeled Case Bank: it’s proprietary to each engagement, compounds with usage, and is cheap to maintain. The effort to stand up the non-parametric version is low (days); the parametric Q-gate is a few more days once you have outcome labels.

Build Your Own (Minimal Recipe)

The 80/20 version skips the parametric Q-function entirely at first.

Components:

1. Embedder — sentence-transformers (or SimCSE as in the paper) to vectorize task descriptions.
2. Case Bank — a vector store (FAISS, Chroma, or even just a list + numpy) holding (task_text, embedding, plan, success_bool).
3. Planner prompt — an LLM call that takes the task + K retrieved cases and returns a subtask list. Put failed cases in too, labeled “this approach FAILED, avoid it.”
4. Executor — an LLM with tool access. Use the real MCP if you’re in that ecosystem, or just function-calling.
5. Retain step — after each task, judge success (LLM-as-judge or ground truth) and append the case.

Build order:

1. Ship the non-parametric loop first: cosine retrieval, K=4, no learning. This is a working continual-learning agent.
2. Add the failure-storage trick — it’s free and meaningful.
3. Only then, once you have a few hundred logged outcomes, train the parametric gate: a nn.Linear-stack MLP on [query_emb ⊕ case_emb] → p(useful) with binary cross-entropy. That’s ~20 lines of PyTorch. Swap retrieval from cosine-top-K to Q-top-K.

The 1–2 genuinely hard parts:

• Reward labeling. You need a reliable “did this task succeed?” signal to train the gate and to label cases. On benchmarks it’s ground-truth; in production it’s an LLM judge or user feedback, and noise here directly poisons the Q-function. This is the real engineering cost.
• Memory curation. K=4 optimal means a noisy/bloated Case Bank hurts. You’ll want dedup (the paper stores only the final-step state per trajectory) and eventually forgetting/eviction.

Reach for: sentence-transformers, faiss, the MCP Python SDK, a frontier model for planner/executor, ~30 lines of PyTorch for the gate.

How to Improve It

1. Beat the plateau with active case generation. The Case Bank saturates at ~3k tasks. Instead of passively logging, generate hard/novel synthetic tasks targeting regions where the Q-function is uncertain (high-entropy retrieval). This turns “warm-up then flat” into directed curriculum learning.
2. Learn forgetting, not just retrieval. They learn which cases to read but keep writing everything. Add a learned eviction/decay policy (MemoryBank-style Ebbinghaus decay, but Q-aware): drop cases whose presence lowers downstream success. Directly attacks the “small memory is better” finding.
3. Make the executor a CBR agent too. Right now only the planner has memory; the executor is stateless. A per-tool case bank (“last time crawl4ai failed on this site shape, I switched to the search summary”) could cut wasted tool calls — and the paper’s own analysis shows tool calls dominate cost at high difficulty.
4. Richer reward than binary. Cross-entropy on 0/1 throws away partial credit. Use the PM (partial-match) score or a step-level reward so the gate learns degrees of usefulness, likely sharper retrieval with fewer cases.
5. Cross-client / federated case banks. For a services business: a shared, anonymized base layer of cases plus per-client private layers. Learn a domain-transfer gate that decides when a generic case is safe to apply — the OOD results suggest cases transfer, so this could compound value across engagements without leaking data.
6. Test on weaker/open backbones. All results use frontier proprietary models. A real contribution would be showing the memory layer rescues a Llama-class model — that would make it a cost lever, not just an accuracy lever.

Glossary

• Case-Based Reasoning (CBR) — solving a new problem by retrieving and adapting solutions to similar past problems; the paper’s core paradigm.
• Case Bank — the agent’s growing external memory; each entry is (task state, plan/action, success reward).
• M-MDP (Memory-Based MDP) — a Markov Decision Process extended with an explicit memory space, so “what to recall” becomes part of the formal policy.
• Retrieval policy (μ) — the learned rule that decides which stored case to pull given the current task; the only learned component.
• Q-function — in RL, an estimate of the value of taking an action in a state; here, the predicted usefulness p(success | state, recalled case).
• Soft / max-entropy Q-learning — RL that maximizes reward plus an entropy bonus, keeping the policy from collapsing to a single deterministic choice (encourages diverse retrieval).
• Temporal-Difference (TD) learning — the standard way to learn a Q-function by nudging it toward reward + discounted next-state value.
• Parametric vs. non-parametric memory — parametric = learns a Q-network to score cases; non-parametric = just appends cases and retrieves by cosine similarity (no training).
• Catastrophic forgetting — when fine-tuning a model on new data erases previously learned skills; a key reason to avoid weight updates.
• Planner–executor architecture — one LLM decomposes the task into subtasks; a second LLM executes each via tools.
• MCP (Model Context Protocol) — a standardized, model-agnostic interface for an LLM to discover and call external tools/data sources.
• SimCSE — a contrastive method for producing sentence embeddings; used here to vectorize task states.
• Pass@3 — metric: counts a task correct if any of 3 attempts succeeds.
• PM (Partial Match) — semantic-similarity score between generated and gold answers, judged by an LLM, more forgiving than exact match.
• OOD (out-of-distribution) — test tasks drawn from a different distribution than the cases collected during training; tests generalization.
• GAIA / HLE / SimpleQA / DeepResearcher — the four benchmarks: long-horizon tool use / frontier expert reasoning / single-hop facts / live web research.