MemGPT: Towards LLMs as Operating Systems

TL;DR

LLMs choke the moment a conversation or document outgrows their context window — a few dozen messages or a short PDF and you’re out of room. Naively growing the window is quadratically expensive and, worse, models get worse at using the middle of a huge context anyway. MemGPT borrows the oldest trick in systems engineering — virtual memory paging between RAM and disk — and gives the LLM a small set of function calls so it can decide, on its own, what to keep in-context (RAM) and what to evict to an external store (disk), then retrieve it later when relevant. The result: a plain GPT-4 with MemGPT roughly triples its accuracy on a deep-memory-recall conversation task (32% → 93%) and is the only system tested that can do multi-hop retrieval beyond a couple of nesting levels. The contribution is not a new model — it’s a control loop and a memory hierarchy wrapped around an unmodified LLM.

Problem & Motivation

The concrete pain: an LLM is a stateless function with a hard input cap. Every message, every document chunk, every prior turn has to fit inside that cap (8k–128k tokens depending on the model) or it simply doesn’t exist for the model. For a conversational agent this means it forgets what you said an hour ago. For document analysis it means a 10-K annual report (often a million tokens) can’t be read at all.

Two “obvious” fixes both fall short:

1. Just make the context window bigger. Self-attention is O(n²) in sequence length, so doubling context quadruples compute and memory. Even if you pay that bill, the “lost in the middle” finding (Liu et al. 2023a) shows long-context models reliably recall information at the start and end of the window but lose track of the middle. You’re paying quadratically for diminishing — sometimes negative — returns.
2. Retrieval-augmented generation (RAG). Bolt a vector search on the front: fetch top-K chunks, stuff them in the prompt, answer. This is one-shot and passive. The model gets exactly one retrieval, can’t decide it needs more, can’t page through results, can’t follow a chain (“the value I found is itself a key — look that up too”). If the gold document isn’t in the first K, it’s gone.

The insight: this is exactly the problem operating systems solved in the 1960s. Programs that needed more memory than physically existed got the illusion of unlimited memory via virtual memory — the OS pages data between fast RAM and slow disk, faulting pages back in on demand. MemGPT asks: what if the LLM itself were the OS process that decides what to page?

What’s New (Core Contribution)

• Virtual context management as a first-class technique. Before: context was a fixed budget you either fit into or truncated. Now: context is a managed resource with tiers (RAM-like main context, disk-like external storage) and the model itself moves data between them. The “infinite context” is an illusion maintained by paging, exactly like virtual memory.
• Self-directed memory via LLM function calls. Before: RAG retrieved once, externally, before the model ran. Now: the LLM generates function calls (archival_storage.search, working_context.append, working_context.replace) as part of its normal output, deciding when and what to read/write. Memory management is a learned behavior driven by the prompt, not a fixed pipeline.
• OS-style control flow: interrupts, memory-pressure warnings, and function chaining. Before: one prompt → one completion. Now: events (user messages, system alerts, timers) trigger inference; a request_heartbeat=true flag lets the model chain multiple function calls before yielding control — enabling genuine multi-step retrieval. The system warns the model when context is ~70% full (“memory pressure”) so it can save important data before eviction.
• Two new benchmarks that prior context tricks can’t fake: Deep Memory Retrieval (DMR) for conversational consistency, and Nested Key-Value retrieval for multi-hop lookups.

What’s genuinely new is the control loop and the self-management, not any individual piece. Vector DBs, function calling, and summarization all pre-existed. MemGPT is the orchestration pattern that turns them into an OS.

How It Works (Technically)

Think of MemGPT as a tiny operating system whose only “CPU” is an LLM. There’s no new math here — no loss function, no training. The cleverness is entirely in how the prompt is structured and how an event loop wraps around the model. Let’s walk the parts.

The memory hierarchy (two tiers)

Main context = the LLM’s prompt tokens. This is “RAM” — the only thing the model can actually see during inference. It’s split into three contiguous regions:

External context = everything outside the window. This is “disk.” Two stores:

• Recall storage — the full message database (every turn ever, searchable).
• Archival storage — an arbitrary-length read/write text DB (in the paper: PostgreSQL + pgvector + an HNSW index for sub-second approximate vector search). This is where uploaded documents and long-term facts live.

The control loop

Everything is event-driven. An event is any input: a user message, a system alert (“document upload complete”), a login notification, or a scheduled timer (which lets the agent act unprompted). Here’s the cycle:

flowchart TB
  E[Event: user msg / system alert / timer] --> P[Parser: event to plain text]
  P --> Q[Queue Manager: append to FIFO queue]
  Q --> CC[Concatenate main context: system + working + FIFO]
  CC --> LLM[LLM Processor: inference]
  LLM --> OUT[Completion tokens]
  OUT --> FE[Function Executor: parse and validate call]
  FE -->|valid call| EXEC[Run function: read/write memory]
  EXEC --> FB[Feed result + any error back into context]
  FB --> HB{request_heartbeat true?}
  HB -->|yes| LLM
  HB -->|no / yield| WAIT[Wait for next event]
  FE -->|reply to user| WAIT
  Q --> MP{Tokens over 70% warn?}
  MP -->|yes| WARN[Insert memory-pressure system message]
  WARN --> LLM
  MP -->|over 100% flush| FLUSH[Evict 50%, recursively re-summarize]

Step-by-step trace — user asks the agent “Who won the first Nobel Prize in physics?” against a Wikipedia archive:

1. The message arrives as an event. The parser turns it into text; the queue manager appends it to the FIFO queue and writes a copy to recall storage.
2. Main context (system instructions + working context + FIFO queue) is concatenated into one string and sent to the LLM.
3. The LLM doesn’t know the answer in-context, so its completion is a function call: archival_storage.search("nobel physics").
4. The function executor parses and validates the call, runs the vector search, and pastes the paginated results back into main context (“Showing 10 of 124 results (page 1/13)…”). Because the model set request_heartbeat=true, control returns immediately to the LLM rather than waiting for the user.
5. The model reads page 1, doesn’t find it, issues archival_storage.search("nobel physics", page=2). Now on page 2 it sees “The 1901 Nobel in physics was awarded to Wilhelm…”.
6. The model yields a normal text reply to the user: “Wilhelm Conrad Röntgen.” That, plus all intermediate function I/O, is written to recall storage.

The “paging through results” in steps 4–5 is the whole point — a passive RAG system would have answered (wrongly) from page 1.

The memory-pressure mechanism (the “page fault” analogue)

This is the part worth slowing down on, because it’s what makes the illusion hold without human intervention.

• Each model has a warning token count (e.g. 70% of the window) and a flush token count (100%).
• When the concatenated prompt crosses the warning threshold, the queue manager injects a system message: “memory pressure — you are about to lose old messages.” This is the model’s chance to call working_context.append(...) or archival_storage.insert(...) to save anything important from the FIFO queue before it’s evicted.
• When it crosses the flush threshold, the queue manager evicts a chunk (e.g. 50% of the window) of old messages and produces a recursive summary: new_summary = summarize(old_summary + evicted_messages). The evicted raw messages aren’t lost — they stay in recall storage and can be searched back in — but they leave the prompt, freeing space. The recursive summary lives in slot 0 of the FIFO queue so the model always has a gist of what fell off the edge.

“Recursive” here just means each summary folds in the previous summary, so the gist compounds over time rather than being recomputed from scratch (which would be impossible — the raw history no longer fits).

The algorithm, simplified

# The MemGPT event loop. The LLM is an unmodified model; all the magic is
# in the system prompt (function schemas) and this orchestration around it.

def memgpt_step(event, main_ctx, fifo, recall_db, archival_db):
    fifo.append(parse(event))            # queue manager: ingest event as text
    recall_db.write(event)               # every turn persisted to "disk"

    while True:                          # function-chaining loop
        manage_pressure(fifo, main_ctx)  # inject warning / flush + re-summarize

        prompt = main_ctx.system + main_ctx.working + fifo.render()
        out = llm(prompt)                # one inference pass -> completion tokens

        call = parse_function_call(out)  # MemGPT interprets output AS a call
        if call is None:                 # plain text -> reply to user, yield
            recall_db.write(out)
            return out                   # control returns to user / next event

        try:
            result = execute(call, main_ctx, recall_db, archival_db)
        except ContextFullError as e:
            result = str(e)              # errors are fed back so model self-corrects

        fifo.append(result)              # function output goes back into context
        recall_db.write(call); recall_db.write(result)

        if not call.request_heartbeat:   # no heartbeat -> yield, wait for event
            return None
        # heartbeat=True -> loop again immediately (multi-step retrieval)


def manage_pressure(fifo, ctx, warn=0.70, flush=1.00):
    used = token_count(ctx) / ctx.window
    if used >= flush:                                  # "page out": evict + compress
        evicted = fifo.pop_oldest(fraction=0.5)
        fifo.summary = summarize(fifo.summary + evicted)   # recursive summary
    elif used >= warn:
        fifo.append(system_msg("MEMORY PRESSURE: save important data now"))

That’s essentially it. No gradients, no fine-tuning. A capable instruction-following base model (the paper shows GPT-4 is the sweet spot; GPT-3.5’s weaker function-calling makes it stumble) plus this loop equals an agent with effectively unbounded memory.

Schematic of MemGPT's two-tier memory and paging. Watch the FIFO queue fill up; at the warning line the model gets a memory-pressure alert, and at the flush line old messages are evicted to recall storage and folded into the recursive summary. This illustrates the mechanism, not the paper's measured token counts.

The function-chaining loop for nested KV retrieval: each lookup returns a value that is itself the next key, so the agent chains searches (heartbeat=true) until a query returns a non-key. Step through to see why fixed-context models hit 0% past a couple of nesting levels.

Built on Prior Work

The honest framing: MemGPT’s main technical contribution, in the authors’ own words, is “a hierarchical tiered memory that uses a long-context LLM as the implementation of main memory.” Long-context architectures and RAG are complements, not competitors — a bigger window just means a bigger “RAM.”

Results & Evidence

Deep Memory Retrieval (DMR) — conversational consistency. The agent is asked a narrow question that can only be answered from a conversation 5 sessions ago. Baselines see a lossy summary of past sessions; MemGPT must search the full history via paginated recall.

A 2.5–3× jump, and notably the stronger base model benefits more because it follows the function-calling protocol more reliably.

Conversation openers — engagement. MemGPT-generated openers (drawing on accumulated persona knowledge) match or slightly beat human-written openers on similarity metrics. Storing facts in working context is what makes this work.

Document QA (NaturalQuestions-Open over Wikipedia). Fixed-context baselines plateau at the retriever’s ceiling: if the gold doc isn’t in the top-K that fits the window, they never see it. MemGPT pages through results and is “unaffected by increased context length.” Caveat the authors own up to: the task is hard for everyone because embedding search often buries the gold doc, and MemGPT sometimes stops paging too early.

Nested KV retrieval — multi-hop. Values can be keys, requiring chained lookups. GPT-3.5 hits 0% at 1 nesting level; GPT-4/Turbo hit 0% by 3 levels. MemGPT with GPT-4 is “unaffected by the number of nesting levels.” This is the cleanest demonstration of function chaining.

What the evidence does and does NOT establish:

• Does establish: self-directed paging + chaining beats one-shot RAG and summarization on memory-recall and multi-hop tasks, and the effect is large.
• Does NOT establish: that it works with weak models (GPT-3.5 with MemGPT is shaky — “limited function calling capabilities”). The whole approach is hostage to the base model’s instruction-following.
• Small eval sets (50 doc-QA questions; one synthetic KV task; an augmented MSC dataset). No latency/cost accounting — every page-through is another full LLM call, which in production is real money and seconds of latency. No comparison against a strong long-context model (e.g. Claude-100k) actually using its window. No measure of how often the model evicts something it later needed.

How You’d Use It

For an AI services shop, MemGPT is less a product and more a pattern you can sell as a capability. Concrete slots:

• Long-lived assistants / “companions” / support agents. Any client wanting an agent that remembers a customer across weeks of interactions. The working-context + recall-storage split is exactly “persistent CRM-style memory the agent maintains itself.” This is the highest-value, lowest-novelty use — and the most defensible, because the value compounds with conversation length.
• Document intelligence over corpora too big to stuff in a prompt. 10-Ks, contract repositories, knowledge bases. The “agent that pages through a Postgres+pgvector store and chains queries” is a clean offering, and the multi-hop chaining is a genuine differentiator over vanilla RAG when answers require collating across documents.
• Memory layer inside a multi-agent system (ARC MAS territory). Each agent gets self-managed long-term memory; the recursive-summary + recall-storage pattern gives agents a shared, searchable history without blowing every agent’s context. The event/heartbeat control flow maps naturally onto inter-agent message passing.

The pragmatic reality in 2026: this paper became the product Letta (formerly MemGPT). For most client work you’d reach for Letta or a framework’s memory module rather than reimplement. The value of understanding the paper is knowing what the memory layer is actually doing so you can debug it, tune the warning/flush thresholds, and explain the cost model to clients.

Build Your Own (Minimal Recipe)

You can stand up an 80% version in a day or two:

Components

1. A capable function-calling LLM (GPT-4-class or better; this is non-negotiable — the paper shows weak models fail).
2. A system prompt that (a) describes two memory tiers and (b) declares 4–5 tools: core_memory_append, core_memory_replace, archival_insert, archival_search(query, page), recall_search(query, page).
3. A vector store — pgvector, Chroma, or even FAISS for a prototype — for archival/recall.
4. An event loop (the pseudocode above): ingest event → render prompt → call LLM → if output is a tool call, execute and feed back → repeat on heartbeat, else reply.
5. A token counter + summarizer for the memory-pressure / flush logic.

Build order

1. Get the bare loop working: LLM emits a tool call, you execute it, you feed the result back. (This is the only part that must be right.)
2. Add archival search with pagination — prove function chaining by making it page to find an answer.
3. Add the FIFO queue + token counting + the warning/flush thresholds with recursive summarization.
4. Add working context (always-on facts) last; it’s the easiest.

The 1–2 genuinely hard parts

• Prompt engineering the system instructions. Getting the model to reliably (a) recognize memory pressure, (b) page instead of hallucinating, and (c) stop paging when done is fiddly and model-specific. This is where most of your time goes.
• The flush/eviction policy. Evict too eagerly and you lose context the model needed this turn; too late and you overflow. The recursive summary quality directly bounds quality.

Reach for: OpenAI/Anthropic function calling, pgvector + an embedding model (text-embedding-3 class), tiktoken for counting. Or just use Letta and skip steps 1–5.

How to Improve It

1. Smarter eviction than FIFO. FIFO + memory-pressure is the OS analogue, but OSes moved past FIFO to LRU/clock algorithms for a reason. Score messages by predicted future relevance (recency × importance × access frequency, à la Generative Agents) before evicting. Testable: hold out questions whose evidence is mid-history and measure recall vs. FIFO.
2. Cache / reduce the paging cost. Every page-through is a full inference. Batch retrieval (return more results per call), or use a cheap model to pre-filter pages and only escalate to the expensive model for the final answer. Measure cost-per-correct-answer, the metric the paper omits.
3. Learn the memory policy instead of prompting it. Fine-tune or RL a small model specifically on when to page / save / evict, using task success as reward. The paper’s GPT-3.5 failures suggest the bottleneck is the policy, not the storage — a dedicated, cheap “memory controller” model could outperform prompting GPT-4 to do it.
4. Detect and prevent destructive eviction. Add a verification step: before flushing, the model (or a verifier) checks whether anything being evicted is referenced by recent goals. Test by injecting “you’ll need X later” early and measuring whether X survives.
5. Multi-agent shared memory tier. Extend external context into a shared archival store with access control, so a team of agents reads/writes common memory. This is the obvious bridge from single-agent MemGPT to a MAS — and a place to test whether self-directed paging causes write conflicts.

Glossary

• Context window — the maximum number of tokens an LLM can take as input at once; its entire working memory per call.
• Virtual memory / paging — OS technique giving programs the illusion of more memory than physically exists by moving data between RAM and disk on demand.
• Main context — MemGPT’s term for the prompt tokens (the “RAM”): system instructions + working context + FIFO queue.
• External context — everything outside the window (the “disk”): recall storage + archival storage.
• Working context — a small read/write text block in the prompt for always-on facts (persona, user preferences).
• FIFO queue — the rolling message history inside the prompt; oldest messages get evicted first.
• Recall storage — the complete, searchable database of every message ever exchanged.
• Archival storage — an arbitrary-length read/write text DB (Postgres + pgvector here) for documents and long-term facts.
• Recursive summary — a running summary that folds the previous summary plus newly evicted messages into a new gist; how MemGPT remembers what fell off the edge.
• Memory pressure — a system warning injected when the prompt nears its token limit, prompting the model to save important data before eviction.
• Function chaining / heartbeat — request_heartbeat=true returns control to the model after a tool call so it can issue another, enabling multi-step retrieval before replying.
• Function calling — an LLM emitting a structured call (name + args) that the host executes; here, the calls are memory operations.
• RAG (Retrieval-Augmented Generation) — fetching external text and adding it to the prompt before generation; MemGPT generalizes this to active, iterative retrieval.
• HNSW — Hierarchical Navigable Small World, an index for fast approximate nearest-neighbor (vector) search.
• Lost in the middle — the finding that long-context models recall info at the start/end of the window better than the middle.
• ROUGE-L — an overlap-based metric (longest common subsequence) for comparing generated text to a reference.