Voyager: An Open-Ended Embodied Agent with Large Language Models

TL;DR

Most LLM agents solve one task and stop; give them a new world and they start from zero again. Voyager is built to run forever. It wraps GPT-4 in three loops — a curriculum that keeps proposing the next reachable goal, a code-writing loop that drafts, runs, and repairs a JavaScript function until the goal is verified, and a skill library that banks every working function so later tasks can retrieve and reuse it. The only “training” is prompting: no fine-tuning, no gradients, no reward model. Run for 160 rounds in Minecraft and Voyager collects 3.3x more unique items, unlocks the wooden→stone→iron→diamond tool tech tree up to 15.3x faster, and covers 2.3x more map than ReAct, Reflexion, and AutoGPT baselines — and when dropped into a brand-new world with an empty inventory, it solves unseen tasks the baselines can’t touch at all, because it brings its skill library with it.

Problem & Motivation

Give GPT-4 a single goal (“build a house”) and a code interface, and it can often get there with enough retries. That’s not the hard part. The hard part is what happens after: the agent has no mechanism for deciding what to do next, no memory of what it already figured out, and no way to reuse a solved sub-problem instead of re-deriving it. Every new task starts the reasoning over from a blank prompt.

That’s fine for a benchmark with one fixed task. It’s useless for an open-ended environment like Minecraft, where there’s no end goal, no score to maximize, no fixed curriculum — just a world you’re supposed to keep exploring, and where fighting a zombie today should make fighting a skeleton next week easier because they’re structurally similar problems. Reinforcement learning agents can learn open-ended behavior but need enormous sample counts and struggle to transfer skills across tasks (catastrophic forgetting). Prior LLM-agent work (ReAct, Reflexion, AutoGPT) added reasoning traces and self-critique to a single task loop, but none of them had (a) a way to decide what to attempt next on their own, or (b) a persistent, reusable memory of skills. Drop them into “explore the world and get as many items as possible” and they wander — no baseline in this paper unlocks even the wooden tool tier of Minecraft’s tech tree in 160 iterations.

What’s New (Core Contribution)

• Automatic curriculum — before: task lists are hand-authored or the agent free-associates with no structure; now: GPT-4 itself proposes the next task by reading the agent’s live state (inventory, biome, nearby entities, completed/failed task history) against the standing objective “discover as many diverse things as possible.” It’s in-context novelty search — no RL, no reward function, just a well-posed prompt with a memory of history.
• Skill library as a growing code memory — before: agent-planner papers treat each generated program as disposable; now: every verified program becomes a permanently indexed skill (keyed by an embedding of its own description), retrievable by future tasks, and composable — craftIronPickaxe() can call smeltIronOre() which can call mineBlock(). This is what actually beats catastrophic forgetting: skills don’t get overwritten, they get referenced.
• Iterative prompting with three feedback channels — before: single-shot code generation, or generic “did it work / did it fail” self-reflection (Reflexion); now: Voyager separately feeds back (1) environment feedback (in-game chat messages the code itself emits, e.g. “I cannot make stick because I need 2 more planks”), (2) raw interpreter execution errors (stack traces, undefined-function errors), and (3) a dedicated self-verification GPT-4 call that acts as critic and outputs both a success/fail verdict and a natural-language critique of what to fix. Up to 4 rounds per task before the curriculum cuts losses and moves on.
• Code as the action space, not motor tokens — before: most embodied-LLM work outputs subgoals or low-level control tokens; now: GPT-4 writes actual JavaScript against a Mineflayer bot API, so a “skill” is a real, temporally-extended, composable function rather than a plan string that still needs a separate executor.

How It Works (Technically)

There’s no math to demystify here — Voyager has no loss function, no gradient step, no learned parameters at all. The entire method is a prompted control loop over three GPT-4/GPT-3.5 roles (curriculum agent, action/coding agent, critic agent) plus a vector database. The “mechanism” is the control flow itself, so that’s where the depth goes.

One task traced end to end. Say the agent’s inventory has a stone pickaxe, some iron ore, and a furnace, and it’s just finished “mine iron ore.”

1. Curriculum agent proposes the next task. It’s fed the state (inventory, equipment, nearby blocks/entities, biome, time, health/hunger, position) plus the list of previously completed and failed tasks, and asked — with the standing directive to maximize discovery of diverse things without proposing something currently unreachable — for the next task with a step of reasoning first. Output: “Task: Smelt 4 raw iron.” (Fig. 3 in the paper shows several of these real transcripts.)
2. Skill retrieval. GPT-3.5 turns the task into a short natural-language plan sketch (“to smelt iron you need a furnace and fuel…”), that text plus current environment feedback becomes a query embedding, and the skill library (a vector DB keyed by embeddings of past skill descriptions) returns the top-5 nearest skills — e.g. mineBlock, craftFurnace, smeltItem if they already exist. Retrieval accuracy is empirically strong: 96.5% top-5 (Table A.4).
3. Code generation. The action agent (GPT-4) gets: coding guidelines (“make this function generic and reusable — it will be called by future functions”), the control-primitive API surface (both hand-written helpers like mineBlock/smeltItem/craftItem and raw Mineflayer bot calls like bot.pathfinder.goto), the retrieved skills as few-shot code examples, the previous round’s code plus whatever feedback it got, the current state, and the task — then asked to reason step-by-step and emit one JavaScript function.
4. Execute in the actual Minecraft world (via MineDojo + Mineflayer). This produces two raw signals: environment feedback (any bot.chat() calls the code itself makes — “I cannot make stick because I need 2 more planks”) and execution errors from the JS interpreter (stack traces, “no item named acacia_axe”).
5. Self-verification. A separate GPT-4 call, given the task description and the post-execution agent state, plays critic: outputs Success: True/False and, on failure, a specific critique (“Find and mine an amethyst shard underground”). This is stricter than Reflexion’s self-reflection because it explicitly checks task completion rather than just reasoning about what went wrong.
6. Loop or commit. If verification says success, the code is added to the skill library (indexed by an embedding of its own GPT-3.5-generated description) and the curriculum is asked for the next task. If it fails, the code, environment feedback, execution errors, and critique all get folded back into step 3’s prompt for another round — capped at 4 rounds total, after which the curriculum abandons the task (it can always resurface it later) rather than burning the whole run on one blocker.

That “GPT-4 writes code, we execute it for real, and feed every signal the execution produced back into the next prompt” cycle is steps 3-6, repeated; steps 1-2 are what makes the sequence of tasks open-ended and increasingly efficient instead of a fixed curriculum.

Architecture & data flow

flowchart TD
  S[Agent state: inventory, biome,\nentities, health, position] --> CUR[Curriculum agent GPT-4]
  HIST[Completed / failed tasks] --> CUR
  CUR -->|next task| RET[Skill retrieval\ntop-5 by embedding similarity]
  SKL[(Skill Library\nvector DB)] <-->|query / return| RET
  RET --> GEN[Action agent GPT-4\ngenerates JS code]
  GEN --> EXE[Execute in Minecraft\nvia Mineflayer]
  EXE -->|env feedback + exec errors| GEN
  EXE --> VER[Self-verification GPT-4 critic]
  VER -->|fail: critique| GEN
  VER -->|success| ADD[Add code to Skill Library]
  ADD --> CUR
  VER -.->|4 failed rounds: give up on task| CUR

Schematic skill library as a 3D graph: each node is a verified skill, edges show which earlier skills a newer skill's code calls or was retrieved alongside. Drag to orbit — this is what "compounding" looks like structurally: complexity grows by composition, not by re-deriving from scratch.

The algorithm, simplified

This is close to the paper’s own Pseudocode 1 (Appendix A.1), trimmed to the essential loop:

def voyager(env, curriculum_agent, action_agent, critic_agent, skill_manager):
    state = env.reset()
    while True:  # runs forever — this IS the "lifelong" part
        progress = curriculum_agent.get_exploration_progress(
            curriculum_agent.completed_tasks, curriculum_agent.failed_tasks
        )
        task = curriculum_agent.propose_next_task(state, progress)  # in-context novelty search

        code, env_feedback, exec_errors, critique, success = None, None, None, None, False
        for attempt in range(4):                       # up to 4 rounds of self-repair per task
            skills = skill_manager.retrieve_skills(task, env_feedback)   # top-5 nearest, by embedding
            code = action_agent.generate_code(
                task, code, env_feedback, exec_errors, critique, skills
            )                                            # GPT-4 writes/refines JS given ALL prior signals
            state, env_feedback, exec_errors = env.step(code)           # actually run it in Minecraft
            success, critique = critic_agent.check_task_success(task, state)  # GPT-4 as critic
            if success:
                break

        if success:
            skill_manager.add_skill(code)               # bank it, indexed by embedding of its description
            curriculum_agent.add_completed_task(task)
        else:
            curriculum_agent.add_failed_task(task)       # curriculum can retry it later, doesn't block

The whole trick is: nothing here is a learned function. propose_next_task, generate_code, check_task_success are all just differently-prompted GPT-4/GPT-3.5 calls; retrieve_skills is nearest-neighbor lookup over text-embedding-ada-002 vectors. The “learning” is entirely the accumulation of state in completed_tasks, failed_tasks, and the skill library across iterations of an otherwise-stateless model.

Built on Prior Work

Results & Evidence

Everything is measured in prompting iterations (one curriculum-proposed task attempt), not wall-clock or environment steps — a fair unit since all methods are equally metered on LLM calls, but note that means “efficiency” here is entirely about round-trip count with GPT-4, not compute or dollars.

• Exploration (Fig. 1): 63 unique items discovered in 160 iterations, 3.3x AutoGPT/ReAct/Reflexion (which barely move off zero — the open-ended goal has no shape for them without a curriculum).
• Tech tree (Table 1): wooden tools 15.3x faster than the best working baseline (AutoGPT, 92 iterations, vs. presumably ~6), stone 8.5x, iron 6.4x, and diamond tools unlocked at all only by full Voyager (1/3 runs, 102 iterations) — every baseline and even Voyager-without-skill-library scored 0/3 on diamond. This is the single most telling number: the skill library is what gets you from “iron tier” to “diamond tier,” not more prompting.
• Map coverage (Fig. 7): 2.3x the traversal distance of baselines, which get stuck in local areas.
• Zero-shot generalization (Table 2): reset to a fresh world, empty inventory, task = “craft diamond pickaxe” / “golden sword” / “get lava in a bucket” / “make a compass.” Every baseline scores 0/3 within 50 iterations. Voyager solves all four tasks 3/3, using the skill library it built in the previous world. Tellingly, giving AutoGPT Voyager’s skill library (without any other Voyager machinery) lifts it from 0/3 to 1-2/3 on some tasks — direct evidence the library itself, not just the curriculum or verification, is doing real transfer work.
• Ablations (Fig. 9, §3.4): self-verification matters most — removing it drops discovered items 73%. Swapping the automatic curriculum for a random one drops discovered items 93%. GPT-3.5 in place of GPT-4 for code generation costs 5.7x fewer items — a coding-quality gap, not a prompting-strategy gap.

Relative drop in discovered-item count when each component is removed (schematic, built from the paper's reported percentages). Self-verification and the automatic curriculum dominate — the two modules worth building first if you're cutting scope.

What this does not establish. All baselines were re-implemented by the authors for an embodied setting they weren’t designed for — ReAct/Reflexion/AutoGPT are NLP-task methods retrofitted here, so the comparison somewhat favors the system purpose-built for this environment (that’s disclosed, not hidden, but it’s not an apples-to-apples “best possible ReAct”). There’s no comparison to pixel-input RL/imitation agents (VPT, DreamerV3) — the authors explicitly say that would be unfair since Voyager gets the high-level Mineflayer API “for free,” so we don’t know how it stacks up against methods that solve perception too. Everything is single-environment (Minecraft); there’s no evidence yet the curriculum/skill-library recipe generalizes to a differently-shaped open world or to physical robotics, which the paper itself flags as future work requiring added safety constraints. And GPT-4’s Minecraft knowledge is presumably baked into its pretraining corpus (recipes, item names, game wiki content) — some of Voyager’s “reasoning” is really retrieval from what GPT-4 already knows about Minecraft, which won’t transport to a domain the model hasn’t seen much of.

How You’d Use It

The direct pattern — not “the Minecraft agent” but “curriculum + skill library + execution-feedback loop” — is a genuinely reusable architecture for any agent you run in an environment with a real interpreter/executor and no natural stopping point: browser automation, internal tooling agents, codebase-maintenance agents, ops/DevOps runbooks, or a long-lived assistant that should get measurably better at your stack over time instead of re-solving the same subtask every session.

• Your harness — add a skill library to your agent runtime. The ablation showing AutoGPT + Voyager’s skill library beats plain AutoGPT is the key result to steal: you can retrofit “accumulate and retrieve verified code/procedures” onto your existing agent loop without rebuilding orchestration. This is close to what a well-designed tool/function registry with semantic retrieval already gives you, formalized and made write-as-you-go — every tool call your agent gets right becomes a retrievable asset for the next run.
• Your harness — decouple self-verification from the actor. Most agent builds skip a dedicated verifier and let the actor mark its own homework. The 73% ablation drop is the strongest number in the paper for why that’s a mistake: add a separate “did this actually succeed” check before you cut it for the extra LLM call it costs.
• Your workflows — automatic curriculum maps directly to “next best action” systems. Anywhere you’d otherwise hand-write a task queue for an agent (onboarding flows, QA sweep order, data-cleaning backlogs, a coding agent’s own backlog), a state-conditioned “propose the next task” prompt can replace a static list and adapt to what’s actually been done vs. failed.
• Your automations — long-running unattended agents that compound. A back-office agent that processes documents, triages tickets, or maintains a codebase over months is exactly the “no natural stopping point” shape this architecture targets: the payoff is that iteration 500 is faster than iteration 5 because the skill library did the learning, not a bigger prompt.
• Less directly applicable: the 4-round retry cap and code-as-action-space pattern assume you have a real execution sandbox with structured error output (interpreter tracebacks, application-level status messages) — if your environment can’t cheaply produce that kind of feedback, this architecture has much less to chew on.

Build Your Own (Minimal Recipe)

You can get a meaningful fraction of Voyager’s value without Minecraft, with about a week of focused build time:

1. Pick a real executable environment with a Python/JS API and — critically — an interpreter that throws structured errors and an app-level way to emit “why this failed” messages. Anything with a decent SDK works: a browser via Playwright, a coding sandbox, a set of internal APIs.
2. Skill library = a vector DB + code store. Any of Chroma/Pinecone/pgvector keyed on OpenAI (or local) embeddings of a one-line skill description; the value is just the function source. This is the cheapest piece to build and, per the ablation evidence, one of the two highest-leverage pieces.
3. Action loop: one LLM call that gets (task, last code, env feedback, exec errors, critique, retrieved skills, current state) and returns new code; execute it for real; capture stdout/stderr/exceptions verbatim — don’t summarize them, the paper’s whole point is that raw execution signal is what fixes bugs.
4. Self-verification as a separate call, not folded into the actor. Give it the post-execution state and the task, ask for a binary verdict plus a one-line critique on failure. This is the highest-leverage single piece per the ablations — build it before the automatic curriculum if you’re cutting scope.
5. Automatic curriculum last. State-conditioned “what’s the next reasonable task given where I am and what I’ve tried” prompt, with a hard-coded retry budget (Voyager uses 4) so one blocked task can’t stall the whole run.
6. The one genuinely hard part: designing the control-primitive API layer (Voyager’s hand-written mineBlock/craftItem/smeltItem wrappers around raw Mineflayer calls). Give GPT-4 too-low-level primitives and it drowns in plumbing; too-high-level and it can’t compose anything new. Expect to iterate this API surface more than the prompts themselves.

Skip the second GPT-3.5 “self-ask” context-augmentation and the warm-up schedule (Table A.1, gradually revealing more state fields as tasks complete) for a first pass — they’re measured tuning refinements, not core to the mechanism.

How to Improve It

• Replace GPT-4-as-verifier with a programmatic or hybrid checker where possible. Self-verification is the single highest-leverage component (73% ablation swing) but the paper admits it sometimes fails to recognize success (e.g., spider string as a kill signal). A rule-based checker for state-diffable tasks (inventory deltas), falling back to the LLM critic only for ambiguous/spatial tasks, would cut cost and likely reduce false negatives.
• Give the curriculum a cost/value model, not just novelty. Right now “propose something new and not-too-hard” is entirely vibes-based prompting. A lightweight bandit or success-rate-weighted proposal (favor tasks whose prerequisite skills already exist and succeed reliably) would likely close some of the gap to a hand-designed curriculum while staying automatic.
• Compress the skill library instead of only growing it. Nothing in Voyager ever merges, deduplicates, or refactors skills — mineBlock-adjacent variants could accumulate indefinitely over a truly long run. DreamCoder’s compression step is the natural thing to borrow here.
• Add multimodal perception. The paper flags this explicitly — Voyager is blind (text-only GPT-4 at the time), which is why 3D-structure building needs a human-in-the-loop critic. Wiring in a vision-language model to replace the “human as critic” role for spatial tasks is a clearly scoped next step, and now realistic (multimodal Claude/GPT-4V-class models exist).
• Cross-domain transfer test. Everything is validated in one environment. Taking the exact three-module architecture (curriculum / library / iterative-feedback loop) and dropping it into a second, structurally different environment — a web-automation sandbox, say — would be the real test of whether this is a Minecraft trick or a general agent architecture.

Glossary

• LLM-powered embodied agent — an agent that perceives and acts inside a simulated or physical world (not just chat) by having an LLM decide what to do.
• In-context lifelong learning — the agent gets better over a long run purely by accumulating things in its prompt/memory (skills, task history), with the underlying model never retrained.
• Automatic curriculum — a sequence of tasks generated on the fly by the system itself, rather than fixed in advance by a human.
• Catastrophic forgetting — a classic continual-learning failure where learning task B degrades performance on previously-learned task A; Voyager avoids it because skills are stored as separate, never-overwritten functions rather than baked into shared weights.
• Skill library — Voyager’s persistent store of verified code, each entry retrievable by the semantic similarity of its description to a new task.
• Embedding — a vector representation of text such that semantically similar text has similar vectors; used here (via text-embedding-ada-002) to find relevant skills by nearest-neighbor search.
• Self-verification — a separate LLM call that checks whether a task actually succeeded and explains why if it didn’t, distinct from the agent that attempted the task.
• Environment feedback — in-game/app-level messages the executed code itself emits (e.g., “need 2 more planks”), as opposed to a language-runtime error.
• Execution error — a stack trace or interpreter error from actually running the generated code (undefined function, invalid argument, etc.).
• Chain-of-thought (CoT) prompting — asking the model to write out its reasoning steps before producing a final answer/action, shown to improve accuracy on multi-step problems.
• Tech tree — Minecraft’s progression of craftable tool tiers (wood → stone → iron → diamond), used here as a proxy for compositional skill mastery.
• Zero-shot generalization (in this paper) — dropping the agent into a fresh world instance with an empty inventory and testing it on tasks it never explicitly trained on, while it keeps the skill library it built elsewhere.
• Mineflayer — the JavaScript library providing a programmable bot API for Minecraft that Voyager’s generated code calls into.
• MineDojo — the open-source Minecraft AI research framework (simulator + internet-scale knowledge base) this paper’s environment is built on.