Knowledge Graph Engineering for Multi-Agentic Systems: The Anthropic Playbook

TL;DR

Multi-agent systems have one shared weakness: each agent’s memory dies when its context window closes. When agents must chain facts across documents, or share a world model that outlives a single session, the context window is not enough. This playbook builds the missing layer — a knowledge graph — using nothing but Claude API calls: Haiku extracts typed entities and subject-predicate-object triples per document, Sonnet resolves the messy surface forms (“Edwin Aldrin” and “Buzz Aldrin”) into canonical nodes, a graph library assembles them, and Sonnet answers multi-hop questions by reasoning over a serialized subgraph with edge-level citations. The “training data” for this whole pipeline is a Pydantic schema. On a six-document Apollo corpus it hit perfect precision (1.00) with lower recall (0.38-0.55) — conservative on purpose — for single-digit dollars in extraction cost. The real payoff is where the graph slots into Anthropic’s five agent patterns: shared memory for orchestrator-workers, a grounding layer for evaluator-optimizer, and a durable world model for overnight loops.

Problem & Motivation

You have a pile of unstructured documents and a question that no single document answers: “which vendors are connected to this incident,” “who works with people who worked on project X.” The answer lives in the connections between documents, not in any one of them.

Retrieval-augmented generation (RAG) — the usual grounding tool — retrieves chunks that look similar to your question and stuffs them in the context window. That works when the answer sits in one passage (single-hop). It fails on multi-hop questions, where the answer requires chaining facts from passages that share no words and no semantic similarity with the query or with each other. The bridge entity that links two unrelated documents is invisible to similarity search precisely because it does not resemble the query.

The classical fix is a knowledge graph: entities as nodes, typed relations as edges, so multi-hop reasoning becomes graph traversal. But building one used to mean training a named-entity recognizer on your domain, training a relation classifier, hand-writing entity-resolution heuristics, and maintaining all three as your data drifts. Every domain shift — news to legal contracts, English names to transliterated ones — meant new labeled data, new training, new evaluation. Weeks of work, brittle on arrival.

This pain compounds in multi-agent systems. Anthropic’s own guidance describes agents as “LLMs using tools based on environmental feedback in a loop” and pushes “simple, composable patterns rather than complex frameworks.” But every one of those patterns assumes information fits in a context window or can be fetched by one search call. When five worker agents each see a different slice of documents and a synthesizer must chain a fact from worker 1 with a fact from worker 4 that neither saw together, the context window either overflows or loses the connection. That is the concrete gap this playbook fills.

What’s New (Core Contribution)

This is a playbook, not a research result — its novelty is a reframing plus a recipe, and it is honest about that. Three contributions:

• The classical NLP pipeline collapses into four prompts. Before: three separately trained models (NER, relation extraction, entity resolution), each needing labeled data and per-domain retraining. Now: one Haiku call per document extracts entities + triples via a structured-output schema; one Sonnet call per entity-type clusters surface forms into canonical nodes; a graph library assembles; one Sonnet call answers queries. The Pydantic schema is the training data. Adaptation drops from weeks of labeling to hours of prompt tuning.

• Structured outputs turn the stage-to-stage interface into a type-checked contract. Before: parse free-form text, catch malformed JSON, validate types at runtime — a failure point that scales linearly with corpus size. Now: client.messages.parse(output_format=ExtractedGraph) either returns a valid typed object or raises. No parsing, no silent corruption. This is what makes “prompt-as-training-data” practical at ten thousand documents rather than merely cute at ten. The playbook is explicit that this pipeline “would not have been practical two years ago” — before guaranteed structured outputs existed.

• The graph is positioned as the infrastructure layer under Anthropic’s five agent patterns. Before: the augmented LLM, prompt chaining, routing, orchestrator-workers, and evaluator-optimizer are each described assuming context-window-sized state. Now: the graph is mapped into each as a named role — retrieval source, gate signal, classifier input, shared memory, grounding layer. This map, not the extraction code, is the part most relevant to someone who already runs a multi-agent system.

What is not new: the extraction/resolution/assembly/query pipeline itself comes from Anthropic’s public cookbook; LLM-based entity resolution and the “blackboard architecture” for multi-agent shared state both predate this note. The synthesis — graph-from-prompts as the blackboard, with provenance — is the framing that ties it together.

How It Works (Technically)

The pipeline is four stages, each a Claude API call, with a graph library in the middle and an evaluation loop wrapped around the whole thing. Follow one entity — Buzz Aldrin — from raw text to a cited answer.

Stage 1 — Extraction (Haiku, one call per document). Each document goes to Haiku with a fixed prompt and a fixed output schema. The schema is three small Pydantic models:

• Entity: name, type (a Literal of PERSON / ORGANIZATION / LOCATION / EVENT / ARTIFACT), and a one-sentence description.
• Relation: source, predicate (a short verb phrase), target.
• ExtractedGraph: entities: list[Entity] and relations: list[Relation].

The prompt gives four guidelines, and each one exists to kill a specific failure mode observed in earlier iterations:

1. Extract only entities central to the document, skip incidental mentions — a precision-favoring knob. It trades recall for noise reduction. On a small corpus you might weaken it to “all mentioned entities” to raise recall; on a large corpus you keep it, because every false entity spawns false relations.
2. Write a one-sentence description grounded in this document — this is the disambiguation signal that stage 2 depends on. “Armstrong — first person to walk on the Moon” vs. “Armstrong — jazz trumpeter”: same name, must not merge. The description replaces what a trained classifier would have learned from labeled data.
3. Use short verb phrases as predicates (“commanded,” “launched from”) — keeps the predicate vocabulary traversable. “Was involved with” is too vague to reason over; “commanded” is not.
4. Every relation must connect two extracted entities — a structural constraint that prevents orphaned edges (a relation referencing an entity the model forgot to extract, i.e. a dangling reference).

From the Buzz Aldrin Wikipedia summary, Haiku returns entities like Entity(name="Buzz Aldrin", type=PERSON, description="Apollo 11 lunar module pilot...") and, from the Apollo 11 document, Entity(name="Edwin Aldrin", ...) — the same person under two surface forms. That mismatch is the problem stage 2 solves.

Stage 2 — Resolution (Sonnet, one call per entity type). Raw extraction gives overlapping mentions: “NASA” / “National Aeronautics and Space Administration,” “Edwin Aldrin” / “Buzz Aldrin.” Build a graph directly from these and you get a fractured mess — one real concept split across disconnected nodes.

String similarity (edit distance, Jaccard) handles typos but is helpless on “Edwin Aldrin” vs. “Buzz Aldrin” — zero character overlap, same person. So the playbook groups entities by type and asks Sonnet (stronger reasoning) to cluster them, feeding it the one-line descriptions as context. Output schema: a Cluster with a canonical name and an aliases list; a ResolvedClusters wrapping the list. The prompt’s four hard constraints map to four failure modes:

• Every input name must appear in exactly one cluster → prevents silent entity loss.
• Genuinely distinct entities get a single-element cluster → prevents over-merging (“Gemini 12” folded into “Project Gemini”).
• Descriptions must be used → prevents falling back to surface-form-only matching.
• Canonical = most complete form → downstream consumers see the most informative name.

On Apollo, resolution compressed 24 surface forms to 22 canonical entities, catching “Edwin Aldrin” → “Buzz Aldrin” and “Neil Armstrong” → “Neil Alden Armstrong.” Two failure modes to monitor: an unmatched name that appears in no cluster silently vanishes (fix: fall back to a single-element cluster), and over-merging that collapses a specific entity into a broad one.

Entity resolution as a collapse. Raw surface forms on the left (some with zero character overlap) cluster by Sonnet — using the extraction descriptions as the signal — into canonical nodes on the right. Click a canonical node to see which aliases folded into it. This is the step string-similarity cannot do.

Stage 3 — Assembly + summarization (graph library + selective Sonnet). With a clean alias map, every relation endpoint is rewritten to its canonical form and loaded into a NetworkX MultiDiGraph. Why a multi-directed graph: two entities can be joined by several distinct predicates (“launched from” and “operated by”), and direction matters (“Armstrong commanded Apollo 11” ≠ “Apollo 11 commanded Armstrong”). Each node carries type, source documents, mention count; each edge carries predicate and provenance document.

The Apollo graph: 22 nodes, 34 edges, one connected component. That single component is itself the evidence resolution worked — fragmented islands would mean variants that should have merged did not. Hub nodes (Apollo program, Apollo 11, degree 9 each) are the entities tying the corpus together.

Then summarization, applied selectively (it is expensive — it pools every mention plus the graph neighborhood into one Sonnet call). The natural trigger is degree: summarize the top-k nodes, or nodes with degree ≥ 3. Sonnet synthesizes an EntityProfile — a 2-3 paragraph summary, 3-5 atomic traceable key_facts, and a structured TimeRange. The prompt’s two teeth: resolve contradictions by preferring the most specific claim, and do not invent facts not supported by the excerpts. For the Apollo program hub, summarization produced a profile spanning 1960 conception through 1973, with a time range no single source document contained in full. This is the step that turns a graph of labels into a graph of knowledge.

Stage 4 — Multi-hop querying (Sonnet over a serialized subgraph). The payoff. To answer “which locations are connected to people who flew on Apollo 11,” you serialize a relevant subgraph — the k-hop neighborhood of a seed entity — into triples, and let Sonnet reason over it. k is the coverage/noise dial: k=1 is direct neighbors (fast, misses indirect links); k=2 is the sweet spot for most multi-hop questions; k=3+ grows fast and may overflow the context window (then you filter or summarize first). For Apollo, k=2 from any hub captures nearly the whole graph.

The prompt is deliberately restrictive: “Answer using only the knowledge graph below. Cite the specific edges that support your answer.” This is the mechanism behind everything. Grounded, Claude returns (Armstrong) --[walked on]--> (Moon) — traceable, limited to what the corpus says, explicit about what the graph lacks. Ungrounded, Claude draws on pretraining and produces a plausible essay about crew birthplaces and universities. On a private corpus where Claude has no prior knowledge, only the grounded answer works at all — and its citations can be verified by simple string-matching against the input triples.

The loop that wraps it all. The evaluation harness — change the extraction prompt, rerun the scorer, watch F1 move — is what turns a demo into a production system. It has the exact shape of a self-improving agentic loop: act (extract), observe (score), learn (tune the prompt), repeat. The playbook’s repeated claim: a loop’s intelligence lives in the quality of its environmental feedback, not in the model. A pipeline with a good scorer improves itself; one without drifts.

Architecture & data flow

flowchart LR
  D[Documents] --> EX[1. Extraction<br/>Haiku · per doc]
  EX -->|entities + S-P-O triples| RES[2. Resolution<br/>Sonnet · per type]
  RES -->|alias to canonical map| AS[3. Assembly<br/>MultiDiGraph]
  AS --> SUM[3b. Summarization<br/>Sonnet · hubs only]
  SUM --> G[(Knowledge Graph<br/>nodes · edges · profiles)]
  AS --> G
  Q[Question] --> QY[4. Querying<br/>Sonnet · k-hop]
  G -->|serialized subgraph| QY
  QY --> A[Grounded answer<br/>every claim cites an edge]
  EVAL[Gold-set scorer] -->|F1 feedback| EX
  A -.->|spot-check| EVAL

The assembled Apollo graph in 3D — drag to orbit. Node size scales with degree, so the two hubs (Apollo program, Apollo 11) are visually obvious; the single connected component is the signal that resolution merged every variant. This is what "multi-hop reasoning = graph traversal" looks like: a path of two hops from Buzz Aldrin reaches the Moon through Apollo 11.

The algorithm, simplified

The one central idea is small enough to hold in your head: four stubbed model calls, a graph in the middle, and the grounded-query wrapper that makes answers traceable.

# The whole pipeline. llm_parse(prompt, schema) -> typed object (structured outputs);
# llm(prompt) -> str. G is a directed multigraph. Boring I/O stubbed; the novel parts spelled out.

def build_graph(documents):
    raw_entities, raw_relations = [], []
    for doc in documents:                                  # Stage 1: Haiku, one call per doc
        g = llm_parse(EXTRACTION_PROMPT.format(text=doc), schema=ExtractedGraph)
        raw_entities += g.entities                         # each has .name, .type, .description
        raw_relations += g.relations                       # each has .source, .predicate, .target

    alias_to_canonical = {}                                # Stage 2: Sonnet, one call per type
    for etype in {e.type for e in raw_entities}:
        names = [(e.name, e.description) for e in raw_entities if e.type == etype]
        clusters = llm_parse(RESOLVE_PROMPT.format(entities=names), schema=ResolvedClusters)
        for c in clusters.clusters:
            for alias in c.aliases:                        # descriptions do the disambiguating,
                alias_to_canonical[alias] = c.canonical    # not string similarity

    for r in raw_relations:                                # Stage 3: rewrite endpoints, assemble
        s = alias_to_canonical.get(r.source, r.source)     # fall back to self => no silent loss
        t = alias_to_canonical.get(r.target, r.target)
        G.add_edge(s, t, predicate=r.predicate, source_doc=r.doc)   # provenance rides every edge

    for node in top_k_by_degree(G, k=10):                  # Stage 3b: summarize hubs only (costly)
        G.nodes[node]["profile"] = llm_parse(
            SUMMARIZE_PROMPT.format(excerpts=mentions(node), relations=edges(node)),
            schema=EntityProfile)                          # "prefer specific claims; invent nothing"
    return G

def ask(G, question, seed, hops=2):                        # Stage 4: grounded multi-hop query
    triples = serialize_subgraph(G, seed, hops)            # k-hop neighborhood -> text triples
    return llm(f"Answer using ONLY this graph; cite the edges.\n{triples}\nQ: {question}")
    # ungrounded == llm(question): plausible but not traceable, and worthless on a private corpus

Built on Prior Work

Results & Evidence

The evidence is a six-document toy corpus (Wikipedia summaries: Apollo program, Apollo 11, Neil Armstrong, Saturn V, Buzz Aldrin, Kennedy Space Center). Treat the numbers as a demonstration of the mechanism, not a production benchmark. The playbook itself is candid about this.

• Extraction volume: Haiku pulled 36 raw entities, 34 relations across the six docs.
• Resolution: 24 surface forms → 22 canonical entities; final graph 22 nodes / 34 edges / 1 connected component; density 34/22 = 1.55 (a “healthy middle” — below 1.0 is sparse, above 2.0 is richly connected).
• Quality vs. a hand-labeled gold set: precision 1.00 (everything Haiku extracted was correct), recall 0.38-0.55 (it missed entities the gold set wanted). Apollo 11: raw F1 0.71. Neil Armstrong: raw F1 0.55.
• Cost claim: for 10,000 documents at ~2,000 tokens each, extraction at Haiku rates is “single-digit dollars,” further cut by prompt caching (fixed schema + instructions) and the Message Batches API (50% off, up to 24h latency).

What the evidence establishes: the pipeline runs end-to-end, resolution catches cases string similarity cannot, and the precision/recall tradeoff is tunable by prompt (the “central entities only” instruction is the lever). What it does not establish: nothing about a hard domain (legal, biomedical, financial filings), no scaling numbers beyond the six-doc run, no relation-quality rigor (relations are scored on (source, target) pairs, ignoring predicate wording — an upper bound on recall). The missed entities are honestly diagnosed as scope mismatches (“Saturn V” was extracted from its own article, not the Apollo 11 summary) and correct filtering (“Purdue University” as non-central) — a prompt-tuning problem, not a model failure. The recommended default — high precision, lower recall — is defensible: a wrong entity spawns wrong relations that mislead multi-hop reasoning; a missing entity just yields an incomplete-but-correct graph.

How You’d Use It

For someone running an AI services company with a multi-agent system already built, the graph is the fix for the pain you have already felt — the orchestrator’s context window bloating as workers report back. Concrete slots:

• Shared memory for orchestrator-workers. Instead of piping every worker’s summary through the orchestrator’s window (which grows linearly with worker count), each worker writes entities and relations to the graph and reads only the subgraph it needs. The orchestrator’s context stays small; the shared state lives in the graph, queryable by any agent at any time. This is the single highest-value use for your ARC-style MAS.
• Grounding layer for evaluator-optimizer. The hard part of any evaluator is its basis for judgment — without ground truth it asks “does this look right” instead of “is this right.” Give it graph access and it becomes a fact-checker: a generator claims “Armstrong commanded Gemini 12,” the evaluator queries the graph, finds no such edge but finds (Aldrin) --[flew on]--> (Gemini 12) and (Armstrong) --[commanded]--> (Apollo 11), and returns precise, cited feedback. Claims absent from the graph escalate to a human rather than being silently accepted or rejected.
• Persistent world model for overnight loops. New documents arriving overnight are extracted, resolved against the existing canonical set (not against each other), and their edges added. The agent’s context flushes; the graph does not. “The agent forgets, the repo does not.”
• A client-facing offering. “Turn your document pile into a queryable, cited knowledge graph” is a sellable engagement with a clear demo (grounded vs. ungrounded answer on the client’s private corpus, where only grounded works) and a low build cost. The commercial moat is not the pipeline — it is public — but the evaluation harness and provenance discipline you wrap around it, which is what separates a demo from something a client can trust on a Monday morning.

Use the decision framework before you sell it, though: single-doc QA or simple routing does not need a graph (RAG or a single agent wins). The graph earns its complexity only when agents must chain facts across sources, share structured state outside context windows, or ground judgments in traceable evidence.

Build Your Own (Minimal Recipe)

The 80%-value version is a weekend build. Smallest useful pipeline:

1. Define the three schemas (Entity, Relation, ExtractedGraph) as Pydantic models. This is your “training data.” (~20 min)
2. Extraction loop: for each document, call Claude Haiku with messages.parse(output_format=ExtractedGraph) and the four-guideline prompt. Collect entities + relations. (~1 hour)
3. Resolution: group entities by type, send each group to Sonnet with the descriptions, get back clusters, build an alias → canonical dict. Add the single-element fallback immediately — it is the difference between a graph that loses nodes and one that does not. (~2 hours)
4. Assembly: rewrite every relation endpoint through the alias map, load into a NetworkX MultiDiGraph, attach provenance to each edge. (~30 min)
5. Query: a BFS serialize_subgraph(seed, k=2) that emits triples, then the restrictive “answer using only this graph, cite edges” prompt. (~1 hour)
6. Then, and only then, the evaluation harness: hand-label two representative documents as a gold set; write a scorer that computes precision/recall on entities and (source, target) pairs on relations; wire an alias map so canonical forms the scorer doesn’t recognize don’t tank your recall artificially. Without this loop you are shipping blind.

The two genuinely hard parts: (a) resolution at scale — you cannot feed 10,000 PERSON entities to one prompt, so you need blocking (group candidates by cheap signals — same last name, shared tokens, embedding similarity — via a simple inverted index, no model call; Sonnet then arbitrates within blocks of 50-100); and (b) the evaluation harness discipline — it is unglamorous, easy to skip, and the thing that actually determines whether the graph is trustworthy. Reach for: anthropic SDK with structured outputs, pydantic v2, networkx (fine to a few hundred thousand edges), and — only past that scale — Neo4j/Neptune or three Postgres tables (entities, relations, aliases) with recursive CTEs for traversal. The extraction and resolution code does not change when you swap the persistence layer; that is an infrastructure decision, not a pipeline one.

How to Improve It

Limitations are leverage. Five concrete, testable pushes past the playbook:

• Temporal edges. The EntityProfile already carries a TimeRange; add the same start/end fields to Relation so the graph captures not just what is true but when. Then filter a subgraph by time window before reasoning, enabling “who held this role in Q3 2024?” — impossible for the current atemporal graph. Low schema cost, high query value.
• Confidence-weighted edges. Attach an extraction-confidence signal to each edge, derived from the model’s uncertainty or from cross-document corroboration (an edge seen in three independent documents outranks one from a single source). Let the evaluator weight its fact-checks instead of treating every edge as equally reliable. Test: does weighting reduce false escalations to humans?
• Predicate equivalence classes for scoring. The current relation scorer ignores predicate wording (an upper bound on recall). Build equivalence classes so “commanded” / “led” / “was commander of” count as one relation, and measure how much your reported recall was hiding real predicate errors.
• Hybrid RAG + graph retrieval. The playbook says the two are complementary but doesn’t fuse them. Build a router: single-hop questions → RAG, multi-hop → graph traversal, and hard cases → both, with the LLM synthesizing across the retrieved passages and the serialized subgraph. This is a directly sellable upgrade and a testable A/B against graph-only.
• Graph-of-graphs for multi-team setups. Each team keeps its own domain graph; a meta-graph records the connections between them, enabling cross-team reasoning without merging incompatible schemas. This is the knowledge-graph analogue of a federated multi-agent architecture — and it inherits the same coordination challenges, so treat it as a research bet, not a weekend build.

The deepest limitation to respect: the graph amplifies the quality of the corpus. A biased or incomplete corpus produces a biased or incomplete graph, and no amount of multi-hop reasoning fixes that. The graph moves the basis for agent decisions from model estimation to extracted facts — but the judgment about which facts matter still lives with the agents, and ultimately with the human who designed them.

Glossary

• Knowledge graph — a structured store of entities (nodes) and their typed relationships (edges), with provenance on each edge.
• Entity resolution — merging different surface forms of the same real-world thing (“Edwin Aldrin,” “Buzz Aldrin”) into one canonical node.
• Structured outputs — an API feature that forces the model’s reply to validate against a schema (here Pydantic), returning a typed object instead of free text.
• Triple (S-P-O) — a subject-predicate-object fact, e.g. (Armstrong, commanded, Apollo 11); the atomic unit of the graph.
• Multi-hop reasoning — answering a question that requires chaining facts across several edges, not reading one passage.
• Provenance — the source document (and extraction context) a triple came from; what makes an answer citable and fact-checkable.
• Subgraph serialization — converting a slice of the graph (the k-hop neighborhood of a seed) into text triples that fit in a context window.
• k-hop neighborhood — every node within k edges of a seed entity; k is the coverage-vs-noise dial (k=2 is the usual sweet spot).
• Blocking — grouping entity candidates by cheap signals (shared tokens, same last name) before expensive LLM resolution runs inside each small block.
• Hub node — a high-degree entity (many connections), typically the most important node and the one worth summarizing.
• Alias map — the dictionary mapping every known surface form of an entity to its canonical name.
• MultiDiGraph — a directed graph allowing multiple distinct edges between the same two nodes (needed because a pair can share several predicates, and direction matters).
• Precision / recall — precision = fraction of extracted items that are correct; recall = fraction of the true items that were found. This pipeline runs high precision, lower recall on purpose.
• Grounding layer — using the graph as the ground truth an evaluator checks claims against, turning “does this look right” into “is this edge in the graph.”
• Augmented LLM / orchestrator-workers / evaluator-optimizer — three of Anthropic’s five agent patterns: a tool-and-memory-equipped model; a coordinator delegating to specialist workers; and a generate-then-critique loop, respectively.