Kosmos: An AI Scientist for Autonomous Discovery

TL;DR

Every AI-scientist system before this hits the same wall: an agent can only take so many actions before its context window fills with junk and it loses the plot. Kosmos’s fix is to not keep the reasoning in a single context at all. Instead, parallel data-analysis agents and literature-search agents each do a bounded task, write a summary into a shared structured world model, and that world model — not a chat transcript — is what decides the next round of tasks. This lets Kosmos run ~20 cycles (≈200 rollouts) on one objective, roughly 8x longer-horizon than prior systems, without going incoherent.

The payoff is concrete: independent scientists rated 79.4% of statements in Kosmos reports as accurate, every claim is traceable to either code or a primary paper, and collaborators said a single 20-cycle run did the equivalent of ~6 months of their own research. The paper showcases seven real discoveries (three reproduced unpublished/withheld findings; four are novel contributions) across metabolomics, materials science, connectomics, statistical genetics, proteomics, and transcriptomics.

The honest read: the engineering claim (world model = long-horizon coherence) is the real contribution, but the paper gives almost no implementation detail on the world model itself. Treat this as a strong existence proof and an architecture pattern to copy, not a recipe.

Problem & Motivation

Data-driven discovery is a loop: search the literature, form a hypothesis, run an analysis, look at the result, update the hypothesis, repeat. LLM agents are good at each step — they can write pandas code, read papers, reason across fields. The trouble is the loop.

Here’s the concrete pain. If you build a naive AI scientist as one long agent loop, every tool call’s output (a 400-line stack trace, a 30-row dataframe, an abstract) gets appended to the context. After a few dozen actions the context is mostly stale noise, the model starts contradicting its earlier conclusions, forgets the objective, and the whole run degrades. The METR “length of tasks AI can do” line (ref [8] in the paper) is exactly this ceiling — agents reliably complete short tasks and fall apart on long ones.

Prior systems each dodged the problem by narrowing scope rather than solving coherence:

• Robin (the authors’ own prior system) does lit-search + analysis cycles but has “limited context sharing between its agents” and is tuned for therapeutics only.
• Sakana’s AI Scientist auto-runs ML experiments and writes papers — but only for machine-learning research.
• Google’s AI co-scientist generates hypotheses through reasoning cycles — but never actually runs or analyzes an experiment.
• The Virtual Lab designed real SARS-CoV-2 nanobodies — but has no exploratory data-analysis capability.

If you can’t state the pain in one sentence: no existing agent can stay on-objective long enough to do a real, multi-day investigation across many analyses without its context degrading. Kosmos is an answer to exactly that.

What’s New (Core Contribution)

1. A structured world model as the coordination substrate (the real contribution). Before: agents shared state by stuffing each other’s raw outputs into a growing shared context (or barely shared at all, as in Robin). Now: every agent writes a structured summary of its task result into a persistent world model; the next cycle’s tasks are generated by querying that world model. The chat context never has to hold the whole investigation — the world model does. This is what buys the long horizon.

2. Scaling agent rollouts ~8x while staying coherent. Before: prior systems run a handful to a couple dozen actions before incoherence. Now: ~200 rollouts (166 data-analysis + 36 literature) per run, ~42,500 lines of code (a 9.8x increase over Robin’s code volume), 1,500 full papers read — all aimed at one objective.

3. Total claim traceability. Before: AI-scientist outputs are prose you have to trust. Now: every statement and figure in a Kosmos report links back to either a specific Jupyter notebook (for analysis claims) or a primary-literature citation (for knowledge claims). This makes independent verification mechanical.

4. Domain-general, validated by domain experts. Before: each prior system worked in one field. Now: because the two agents are general-purpose and the world model is domain-agnostic, the same system produced expert-validated findings in six different fields, with three reproducing withheld/unpublished results it could not have seen.

Be precise about novelty: items 2–4 are consequences. The genuinely new mechanism is item 1 — replacing “shared context window” with “shared structured world model as the planning oracle.” That’s the idea worth copying.

How It Works (Technically)

Think of Kosmos as a closed-loop scheduler wrapped around two worker-agent fleets, with a structured world model sitting in the middle as both blackboard and planner.

The inputs. A scientist provides (a) an open-ended research objective in plain English (“Identify the specific metabolic adaptations that enable KOR-induced cerebroprotection during hypothermia”) and (b) a dataset (≤ ~5 GB; tabular/processed, not raw images or FASTQs).

The two worker fleets.

• Data Analysis Agent — a general-purpose computational agent (the BixBench-style agent, ref [5]). Given a narrow task, it writes and runs Python in a Jupyter environment: loads the data, does QC, transforms, runs statistics, makes plots. Output = a notebook + a result summary.
• Literature Search Agent — the PaperQA2-style agent (ref [6]) that reads full-text papers and answers questions with primary-source citations. Output = synthesized findings + citations.

The cycle (this is the heart). One cycle =

1. Plan: Query the world model to propose up to ~10 tasks for this cycle — a mix of analysis tasks and literature tasks, each phrased to advance the objective.
2. Execute in parallel: Spin up parallel instances of the two agents, one per task. Each runs independently and bounded — it does its task and stops. (Bounding each rollout is what keeps any single context small.)
3. Summarize & integrate: Each agent returns a structured summary of what it found. Kosmos writes these summaries into the world model — not the raw notebooks or raw paper text, the distilled findings.
4. Re-plan: Kosmos queries the updated world model to decide the next cycle’s tasks. If the world model says “salvage pathway looks up, but we haven’t ruled out de-novo synthesis,” the next cycle gets a task to test that.

Repeat for up to ~20 cycles / 12 hours. When the world model indicates the objective is satisfied, Kosmos synthesizes 3–4 scientific reports, each statement carrying its provenance link.

Why the world model is the trick. In a normal long agent loop, “what should I do next?” is answered by re-reading the entire growing transcript — which is exactly what degrades. In Kosmos, “what next?” is answered by reading a curated, structured state object that only ever holds distilled findings. The raw 400-line trace from a failed coloc run (which actually happened in Discovery 4) lives in a notebook for traceability but never pollutes the planning context. Coherence is preserved by separating the audit trail (big, in notebooks) from the planning state (small, in the world model).

A real trace (Discovery 1, the one I read in full). Objective: why is hypothermia neuroprotective in mouse brain? Dataset: LC-MS metabolomics, three groups.

• Cycle work: an analysis agent checks group balance and variance, decides a log10 transform is needed, runs differential abundance → flags nucleotides and amino-acid derivatives. Summary → world model.
• Next plan from world model: “nucleotide metabolism is the top pathway; test the salvage hypothesis.” → analysis agent finds precursors down, phosphorylated products up (the salvage signature).
• In parallel: a literature agent confirms nucleotide salvage is a known energy-saving strategy in hypoxia/hypothermia (cites refs [10–13]) → world model.
• World model spots a competing explanation: could this be de-novo synthesis instead? → analysis agent correlates products vs de-novo substrates, finds no relationship, rules it out.
• Synthesis: salvage-pathway engagement. Result matched the human study with R² = 0.998 on the top-15 metabolite fold-changes.

Notice there’s no fancy math here — the “model” is an orchestration design, not a new loss function. The intelligence is in the control flow: bounded parallel tasks → structured distillation → world-model-driven re-planning.

Architecture & data flow

flowchart TB
  subgraph Input
    OBJ[Research objective<br/>plain English]
    DATA[(Dataset &le; ~5GB)]
  end
  OBJ --> WM
  DATA --> WM

  subgraph Cycle [One discovery cycle &times; ~20]
    WM[Structured World Model<br/>distilled findings + open questions]
    WM -->|propose up to 10 tasks| PLAN{Task planner}
    PLAN -->|analysis tasks| DA[Data Analysis Agents<br/>parallel · Jupyter/Python]
    PLAN -->|literature tasks| LA[Literature Agents<br/>parallel · full-text + citations]
    DA -->|structured summary<br/>+ notebook link| WM
    LA -->|structured summary<br/>+ citations| WM
  end

  WM -->|objective satisfied| SYN[Synthesize 3-4 reports]
  SYN --> REP[Reports: every claim links to<br/>a notebook or a primary source]

Schematic: why a shared world model beats a single growing context. Toggle between "naive long context" (signal drowned by accumulating raw tool output) and "world model" (planning state stays small and clean because raw output is offloaded to notebooks). Illustrates the coherence argument, not the paper's measured numbers.

The algorithm, simplified

# Kosmos main loop — the contribution is the world model, not the agents.
# Stubs: analysis_agent(task)/lit_agent(task) -> Result(summary, artifact_link)

def kosmos(objective, dataset, max_cycles=20):
    world = WorldModel(objective=objective, dataset=dataset)  # structured state, NOT a transcript

    for cycle in range(max_cycles):
        # 1. PLAN: tasks come from the curated world state, not a giant chat log
        tasks = world.propose_tasks(k=10)        # mix of analysis + literature, objective-aligned
        if world.objective_satisfied():
            break

        # 2. EXECUTE: each rollout is BOUNDED -> its context never balloons
        results = run_parallel(
            [analysis_agent(t) if t.kind == "analysis" else lit_agent(t)
             for t in tasks]
        )

        # 3. INTEGRATE: write DISTILLED summaries into the world model.
        #    Raw notebooks / paper text stay external (audit trail), out of planning context.
        for r in results:
            world.integrate(summary=r.summary, provenance=r.artifact_link)

    # 4. SYNTHESIZE: every claim carries its notebook or citation link
    return world.write_reports(n=4, cite=True)

The two lines that are the paper: world.propose_tasks() (planning reads curated state, so it doesn’t degrade) and world.integrate(summary=..., provenance=...) (only the distilled finding enters planning state; the raw artifact is kept separately for traceability).

Built on Prior Work

Results & Evidence

Headline numbers.

• 79.4% statement accuracy overall, from 102 statements across 3 reports, judged by independent expert scientists. Broken down: 85.5% of data-analysis statements reproducible, 82.1% of literature statements verifiable to primary sources, 57.9% of interpretation/synthesis statements accurate.
• ~4.1 expert-months per run by a bottom-up tally (15 min/paper, 2 hr/analysis, 174 hr/month).
• ~6.14 expert-months for a 20-cycle run by an orthogonal estimate from seven collaborating academic groups — and they reported value scales with cycles, roughly doubling from cycle 5 → 20, with valuable findings rising near-linearly.
• Seven discoveries: three reproduce withheld/unpublished results (e.g., the metabolomics R²=0.998 match; SOD2/myocardial-fibrosis MR with r=0.9991 vs human analysis on 31 shared proteins), four are novel (e.g., a flippase “pro-phagocytic axis” in aged entorhinal neurons).

What the evidence establishes. That a world-model-coordinated multi-agent system can run far longer than prior systems, produce mostly-accurate, fully-traceable output, and surface real findings across many fields — including reproducing results it provably didn’t have access to. That’s a strong existence proof for the architecture.

What it does NOT establish — read this part.

• No head-to-head ablation of the world model. The central claim (“world model → coherence”) is never tested by removing it and measuring degradation. We’re shown the system works, not that this component is why.
• The 57.9% interpretation accuracy is the real ceiling. Kosmos “tends to make excessively strong claims” and “conflate statistically significant with scientifically valuable.” Nearly half its interpretive statements are off. The value proposition the authors honestly state: not that it’s right, but that broad unbiased exploration surfaces true, interesting things a human then validates.
• Expert-time estimates are self-reported by collaborators, not a controlled study. Useful directional signal, not a benchmark.
• Cherry-picking risk: seven hand-selected discoveries from an unknown number of runs. We don’t see the failure rate or the boring runs.
• Provenance ≠ correctness: every claim is traceable, but a traceable claim can still be wrong (the SOD2/miR-222 binding site didn’t hold up against current databases). Traceability makes errors findable, not absent.
• No implementation detail on the world model’s data structure, schema, or how propose_tasks actually works. You cannot reproduce the core mechanism from this paper.

How You’d Use It

For an AI services company, this paper is a blueprint for a high-value offering: long-horizon, traceable analysis agents for data-rich clients. Concrete angles:

• “Autonomous analyst” engagements. A client hands you a clean tabular dataset and a fuzzy question. You run a Kosmos-style loop and deliver a report where every claim links to a runnable notebook or a citation. The traceability is the differentiator — it’s what makes a non-technical stakeholder trust agent output, and what survives a compliance review.
• The world-model pattern fixes your existing MAS. You’ve built ARC MAS; you know agents go incoherent on long tasks. The actionable lesson: stop sharing raw outputs between agents. Insert a structured shared-state object that holds distilled findings + open questions, and make your planner read that instead of the transcript. Keep raw artifacts (logs, dataframes, traces) in external storage, referenced by ID. This is a retrofittable upgrade, not a rewrite.
• Traceability-as-a-feature. “Every sentence cites its source code or paper” is sellable on its own to regulated clients (pharma, finance, legal) regardless of domain.
• Where it slots in: between “client has data” and “client needs analysis they’d otherwise pay a specialist months for.” It’s a force-multiplier for a single analyst, with a mandatory human-validation step — which is also billable.

Effort to stand up a useful v1: weeks, not months, if you already have analysis + retrieval agents (see next section). The hard part is the world model and the validation discipline, not the agents.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value:

Components

1. A data-analysis agent — an LLM with a sandboxed Python/Jupyter tool. Reach for: an off-the-shelf code-execution agent (e.g., a LangGraph or AI-SDK loop with a Jupyter/e2b sandbox), GPT/Claude as the model.
2. A literature agent — retrieval + full-text QA with citations. Reach for: PaperQA2 (open source, exactly this) or a RAG pipeline over a paper corpus.
3. The world model — the part you actually build. Simplest honest version: a JSON/SQLite object with findings: [{claim, provenance_id, confidence}], open_questions: [...], objective. Two LLM-backed methods: integrate(summary) (merge a new finding, dedupe, possibly raise a new open question) and propose_tasks() (read findings + open_questions → emit next tasks).
4. An orchestrator — the for cycle in range(N) loop with run_parallel.

Build order

1. Get one analysis agent and one lit agent working bounded (single task in, summary + artifact link out).
2. Build the world model as a dumb store first; planner = “pick the oldest open question.” Confirm the loop runs and stays on objective for 5 cycles.
3. Make propose_tasks and integrate LLM-backed so planning actually reasons over accumulated state.
4. Add report synthesis with provenance links last.

The 1–2 genuinely hard parts

• integrate / propose_tasks quality. Deciding what to distill, what counts as a new open question, and when the objective is “done” is the whole game. Garbage distillation → incoherent planning, and you’re back to square one. Expect to iterate heavily here.
• Bounding rollouts without losing necessary context. Each worker needs just enough of the world state to do its task and no more. Too little → it redoes work; too much → you’ve reinvented the bloated context.

You do not need RL, fine-tuning, or anything exotic. This is pure orchestration over capable base models.

How to Improve It

Limitations as leverage — concrete, testable:

1. Add a “scientific taste” critic. The 57.9% interpretation accuracy is the biggest weakness. Add a dedicated critic agent (or a fine-tuned/RL-trained scorer) that grades each proposed interpretation for scientific significance, not just statistical significance, before it enters the world model. Test: does interpretation accuracy rise above 58% with the critic in the loop?

2. Run the missing ablation yourself. Build the world-model version and a naive-shared-context version, run both on the same objective, measure coherence/accuracy vs cycle count. This is the experiment the paper skipped and it’s the most valuable single thing you could publish.

3. Auto-triage discoveries. The paper admits finding the valuable discoveries among 3–4 narratives × 25 claims is a human-time sink. Add a novelty+significance ranker that surfaces the top candidates, so a human validates 5 claims instead of 75. Testable against expert rankings.

4. Let the world model trigger external data pulls. Current Kosmos can’t fetch reference/validation datasets autonomously. Give the world model a “fetch orthogonal dataset” task type wired to public APIs (GEO, UniProt, GWAS catalogs). This directly addresses the inability to self-validate.

5. Mid-run human steering. Currently a scientist can’t nudge between cycles. Expose the world model’s open-questions list as an editable surface so a human can up/down-weight directions mid-run. Cheap to add, likely high value, and turns “12-hour black box” into “interactive co-pilot.”

6. Stochastic-convergence harness. Runs don’t reliably converge on the same discoveries. Run N independent seeds and have the world model surface only findings that replicate across seeds — turning stochasticity from a bug into a built-in robustness filter.

Glossary

• World model (here) — a structured, persistent state object holding distilled findings + open questions for one objective; serves as both shared memory and the planner’s input. Not a learned/predictive model in the RL sense — it’s an orchestration data structure.
• Agent rollout — one bounded run of a worker agent on a single assigned task, from prompt to result summary.
• Cycle — one plan→execute→integrate→re-plan round; Kosmos runs ~20 per investigation.
• Coherence (of an agent) — staying on-objective and self-consistent over many actions; the thing that degrades as context fills with stale output.
• Context window — the bounded token buffer an LLM sees; the root constraint that long agent loops blow past.
• Provenance / traceability — every report claim linked to its source notebook (analysis) or citation (literature), so it can be independently checked.
• Data-analysis agent (BixBench-style) — general-purpose agent that writes and executes Python in a notebook to analyze data.
• Literature agent (PaperQA2-style) — agent that reads full-text papers and answers with primary-source citations.
• Mendelian randomization (MR) — a stats method using random inheritance of genetic variants as natural instruments to infer causal (not just correlational) effects of a biomarker on an outcome. Used in Discovery 4 (SOD2 → myocardial fibrosis).
• Fine-mapping (SuSiE) / colocalization (coloc) — genetics methods to pinpoint which variant in a region is causal and whether two traits share that causal variant. Mentioned because Kosmos’s coloc step failed and it routed around it via fine-mapping — a nice example of agentic robustness.
• Interpretation/synthesis statement — a report claim that connects analysis + literature into a conclusion (vs a raw analysis or raw literature statement); the category where Kosmos is weakest (57.9%).
• Expert-month — the paper’s unit of human-effort-equivalent (174 working hours).