ReAct: Synergizing Reasoning and Acting in Language Models

TL;DR

LLMs had two separate tricks: chain-of-thought (CoT), where the model “thinks out loud” but stays trapped in its own head and hallucinates facts, and action-generation, where the model emits tool calls but can’t plan or reason about what it sees. ReAct’s idea is almost embarrassingly simple: prompt the model to produce both, interleaved — Thought → Action → Observation → Thought → ... — so reasoning steers which actions to take, and the observations from actions feed back into reasoning. On knowledge tasks (HotpotQA, FEVER) this kills hallucination by grounding the model in a Wikipedia API; on long-horizon decision tasks (ALFWorld, WebShop) it beats imitation- and RL-trained agents by +34% and +10% success rate respectively, using only 1-6 in-context examples. The headline isn’t a benchmark number — it’s that the now-ubiquitous “think, then act, then observe, repeat” agent loop comes straight from here.

Problem & Motivation

Before ReAct, the two halves of an agent lived in separate papers and didn’t talk to each other:

Chain-of-thought is a closed black box. CoT prompting (Wei et al., 2022) gets a model to write intermediate reasoning steps before answering — great for arithmetic and logic. But it reasons entirely from frozen internal weights. It can’t look anything up, can’t check a fact, and can’t update mid-stream. So it confidently invents facts (“Apple Remote was designed to control Apple TV”) and then propagates that error through every subsequent step. The pain is concrete: a wrong premise in step 1 poisons the whole chain, and the model has no way to notice.

Action-only agents can’t plan. The other line of work (WebGPT, SayCan, etc.) prompts an LLM to emit actions — search this, click that — usually trained with expensive imitation or reinforcement learning. But these agents map context → action implicitly. They don’t decompose a goal into subgoals, don’t track “what have I already tried,” and don’t reason about an observation before reacting. In ALFWorld, an act-only agent literally tries to take a pepper shaker from a sink basin that doesn’t contain one, over and over, because it never reasons about state.

The one-sentence pain: reasoning without acting hallucinates; acting without reasoning flails — and nobody had shown that combining them in one model is both easy and systematically better.

What’s New (Core Contribution)

1. The interleaved Thought/Action/Observation paradigm. Before: reasoning (CoT) and acting (tool calls) were distinct prompting regimes. Now: a single prompt elicits both, in one trajectory, where thoughts are just another kind of “action” that happens to modify the context instead of the world. This is the actual contribution — and it’s the loop every modern agent framework (LangChain agents, the original AutoGPT, function-calling loops) is built on.

2. Reasoning as a “free” internal action. Before: an action a always touched the environment and produced an observation. Now: the action space is augmented to  = A ∪ L (real actions plus the space of language). A language action — a thought — produces no environment observation; it only updates the model’s context for the next step. This reframing is the conceptual core (more below).

3. Few-shot beats trained agents on decision tasks. Before: interactive agents needed 10³–10⁵ trajectories of imitation/RL training. Now: 1-6 hand-written ReAct exemplars in the prompt outperform them. That’s the result that made people take prompt-based agents seriously.

4. A practical hybrid: ReAct ↔ CoT-SC backoff. Before: you picked internal knowledge (CoT) or external (retrieval). Now: a simple heuristic switches between them — fall back to CoT self-consistency when ReAct can’t answer in N steps, and fall back to ReAct when CoT’s votes are split. Genuinely useful, genuinely simple.

Honest take: contributions 1 and 2 are the same idea viewed two ways, and the “novelty” is conceptual framing rather than a new algorithm. But framing that spawns an entire engineering pattern is real novelty.

How It Works (Technically)

There is no new model, no new loss, no training (for the main results). ReAct is a prompting protocol plus a control loop. Understanding it means understanding one equation and one loop.

The standard agent setup (the baseline). At step t, an agent sees observation oₜ and picks action aₜ from a policy π(aₜ | cₜ), where the context is the full history:

cₜ = (o₁, a₁, o₂, a₂, ..., oₜ₋₁, aₜ₋₁, oₜ)

Plain English: “given everything I’ve seen and done so far, pick the next action.” The hard part is that the mapping from a long messy history to the right action is implicit — it secretly requires multi-step reasoning the model has no scratchpad for. That’s why act-only agents fail.

The ReAct move — augment the action space. Define a new action space:

 = A ∪ L

where A is the real actions (search, click, take) and L is the entire space of natural language. Now the model can emit an action âₜ ∈ L — a thought. Here’s the operationally critical bit: a thought does not affect the environment, so it returns no observation. Instead it just appends itself to the context:

cₜ₊₁ = (cₜ, âₜ)        # a thought only updates the agent's own context

That’s the whole trick. A thought is a “write to working memory” that costs one generation step and grounds the next decision. Contrast with a real action aₜ ∈ A, which does hit the environment and yields oₜ₊₁:

cₜ₊₁ = (cₜ, aₜ, oₜ₊₁)  # a real action updates context AND pulls in an observation

So the trajectory becomes a stream like: Thought₁ → Action₁ → Obs₁ → Thought₂ → Action₂ → Obs₂ → … where some “actions” are thoughts (no obs) and some are real (with obs). Because L is unbounded, you can’t learn this from scratch cheaply — you need a model with strong language priors. The paper uses frozen PaLM-540B prompted with a handful of human-written exemplars; no fine-tuning for the main results.

Two regimes for two task types:

• Reasoning-heavy tasks (HotpotQA, FEVER): dense thoughts — alternate thought/action/observation every step. Reasoning is the bottleneck, so think constantly.
• Action-heavy, long-horizon tasks (ALFWorld, WebShop): sparse thoughts — let the model decide when to inject a thought (it might take 5 actions then think once). Thinking before every low-level “go to drawer 2” would be wasteful noise.

One concrete trace (HotpotQA, “what other device can control the program Apple Remote was designed for?”):

1. Thought: “I need to search Apple Remote and find the program it was designed to interact with.”
2. Action: search[Apple Remote]
3. Obs: “The Apple Remote… originally designed to control the Front Row media center program…”
4. Thought: “Apple Remote controls Front Row. I need to search Front Row next and find what other device can control it.”
5. Action: search[Front Row] → Obs: “Could not find [Front Row]. Similar: […]”
6. Thought: “Front Row is not found. I need to search Front Row (software).”
7. Action: search[Front Row (software)] → Obs: “…controlled by an Apple Remote or the keyboard function keys.”
8. Thought: “So the answer is keyboard function keys.” → Action: finish[keyboard function keys]

Notice what reasoning bought: it decided the second search target from the first observation (act→reason→act), and it recovered from a failed search by reformulating the query (exception handling). A CoT-only model would have guessed “iPhone, iPad, iPod Touch” from a hallucinated premise (it does, in the paper). An act-only model can’t synthesize the final answer because that requires reasoning over the whole trajectory.

The action space they built for QA is deliberately weak — three actions designed to mimic how a human browses Wikipedia and force the model to reason rather than lean on a strong retriever:

• search[entity] → first 5 sentences of the entity’s page, or top-5 similar entities if it doesn’t exist.
• lookup[string] → next sentence on the current page containing string (like Ctrl+F).
• finish[answer] → end and return the answer.

Architecture & data flow

flowchart TD
  Q[Task / Question] --> CTX[Context cₜ: history of thoughts, actions, observations]
  CTX --> LLM[Frozen LLM + few-shot ReAct exemplars]
  LLM --> DEC{Generated token: thought or action?}
  DEC -->|Thought âₜ ∈ L| THINK[Append thought to context<br/>NO environment call, NO observation]
  DEC -->|Action aₜ ∈ A| ENV[Execute in environment<br/>search / lookup / click / take]
  THINK --> CTX
  ENV --> OBS[Observation oₜ₊₁]
  OBS --> CTX
  DEC -->|finish action| OUT[Final answer / task done]

Step through a ReAct trajectory vs. an Act-only trajectory on the same task. Watch how "thoughts" (which never call the environment) let the agent reformulate after a failed search, while the act-only agent loops on the same dead-end. Schematic, built to mirror the paper's Figure 1 example.

The algorithm, simplified

# The entire ReAct control loop. The "intelligence" is in the prompt exemplars,
# not in this code. llm() -> str is a frozen model; env.step(action) -> observation.

def react(task, exemplars, env, max_steps=7):
    # exemplars: a few hand-written Thought/Action/Observation trajectories.
    # They teach the format AND the reasoning style by example only.
    context = exemplars + f"\nQuestion: {task}\n"

    for step in range(max_steps):
        # Model generates ONE line: either "Thought: ..." or "Action: ..."
        gen = llm(context + "Thought:")        # ask for a thought first (dense mode)
        context += f"Thought:{gen}\n"

        action = llm(context + "Action:")      # then the action it implies
        context += f"Action:{action}\n"

        if action.startswith("finish"):
            return parse_answer(action)        # done — return the synthesized answer

        # A REAL action hits the world and returns an observation.
        # (A pure thought would have no env call — here we always pair them in dense mode.)
        obs = env.step(action)                 # search[...] / lookup[...] / click[...]
        context += f"Observation: {obs}\n"     # observation feeds the NEXT thought

    return backoff_to_cot_sc(task)             # ReAct ↔ CoT-SC heuristic on timeout

The few-shot exemplars do all the heavy lifting: they show the model what a good thought looks like (decompose the goal, extract from the observation, reason commonsensically, reformulate on failure, synthesize the answer). No special format engineering — annotators just typed their reasoning on top of their actions.

Built on Prior Work

The honest lineage: ReAct sits exactly between CoT (reason, don’t act) and the policy-LLM line (act, don’t reason), and its closest cousin is Inner Monologue. The delta over IM — using real reasoning rather than parroted state — is the ablation they’re proudest of.

Results & Evidence

Knowledge tasks (PaLM-540B, prompting):

• HotpotQA exact-match: Standard 28.7, CoT 29.4, ReAct 27.4, ReAct→CoT-SC 35.1 (best). ReAct alone slightly loses to CoT here.
• FEVER accuracy: Standard 57.1, CoT 56.3, ReAct 60.9, CoT-SC→ReAct 64.6 (best). ReAct wins because fact verification needs up-to-date retrieval.
• The combined ReAct+CoT-SC methods reach 21-sample CoT-SC quality with only 3-5 samples — a real efficiency win.

Why the failure analysis matters more than the EM scores. Hand-labeling 200 trajectories: CoT’s false-positive (confidently wrong) rate is 14% vs ReAct’s 6%, and hallucination is 56% of CoT’s failures vs 0% for ReAct. The flip side: ReAct’s structural rigidity causes more reasoning errors (47% vs 16% of failures), including a nasty failure mode where it loops, repeating the same thought/action — and 23% of ReAct failures come from uninformative searches derailing it. This is the honest trade: grounding vs. flexibility.

Decision-making tasks (the headline result):

• ALFWorld: ReAct best-of-6 = 71% success vs Act 45% vs BUTLER (imitation-learning, trained on 10⁵ trajectories) 37%. Even ReAct’s worst trial (48%) beats everyone else’s best. ReAct vs ReAct-IM (Inner-Monologue style): 71% vs 53%.
• WebShop: ReAct success rate 40.0% vs IL 29.1% vs IL+RL 28.7% — an absolute +10% over methods trained on ~10⁴ instructions, using one-shot prompting.

Fine-tuning: when you fine-tune small models on 3,000 ReAct trajectories, ReAct goes from worst (prompting an 8B model can’t learn both reason+act from few-shot) to best — a fine-tuned PaLM-8B ReAct beats all 62B prompting methods, and 62B ReAct beats all 540B prompting methods. Acting-style methods generalize because they teach “how to find info,” not “memorize facts.”

What the evidence does NOT establish:

• Still far below domain-specific SoTA (HotpotQA supervised SoTA ~67 EM vs ReAct ~35) and below expert humans on WebShop (59.6% SR). ReAct is a general method, not a benchmark crusher.
• Everything rides on PaLM-540B; the in-context learning version needs a very capable base model. (GPT-3 appendix results are even better, but it’s still frontier-model-dependent.)
• The deliberately-weak Wikipedia API means absolute QA numbers are artificially low — they chose interpretability over raw retrieval power.
• Long-horizon tasks blow past context limits if you need many demonstrations — a real ceiling on the pure-prompting version.

How You’d Use It

If you run an AI services company, you are already shipping ReAct whether you call it that or not — every “tool-using agent” with a reason/act/observe loop is this paper. Knowing the original sharpens how you build and sell it:

• Grounded RAG-as-agent. Instead of one-shot “retrieve then answer,” let the agent reason about what to retrieve next based on what it just read (the act→reason→act pattern). This is the difference between a brittle RAG demo and a system that handles multi-hop client questions (“which of our contracts expire before the vendor’s renewal date?”). The failure analysis tells you the ROI directly: it slashes confident-but-wrong answers, which is exactly the liability clients fear.
• The interpretability sell. ReAct trajectories are human-readable: a client can read the thought stream and see why the agent did something, and even edit a thought to correct it mid-run (the paper shows thought-editing as a control mechanism). For regulated or high-trust clients, “you can audit and steer the reasoning” is a concrete, billable differentiator over a black-box answer.
• In your MAS work specifically: ReAct is the intra-agent loop; your multi-agent orchestration is the inter-agent layer. Each worker agent should run a ReAct loop with its own scoped toolset, and the sparse-vs-dense-thought distinction maps onto agent roles — a planner agent wants dense thoughts, a low-level executor wants sparse ones. The “thought = context write with no env effect” framing is also a clean way to design agent memory: thoughts are the cheap writes, tool calls are the expensive ones.
• The hybrid backoff is a cheap reliability upgrade. Wire in the ReAct↔CoT-SC heuristic: if the agent can’t ground an answer in N tool calls, fall back to pure model knowledge with self-consistency voting, and vice versa. Few lines of orchestration, measurable accuracy bump.

Build Your Own (Minimal Recipe)

You can build a working ReAct agent in an afternoon with any function-calling-capable model.

Components:

1. A capable LLM (GPT-4o / Claude / a strong open model — weaker models won’t reliably interleave).
2. A tool registry: 3-5 functions with crisp text signatures (search(query) -> str, lookup(s) -> str, finish(answer)).
3. A prompt with 1-6 hand-written exemplar trajectories in Thought:/Action:/Observation: format. This is where 80% of the quality lives — write good thoughts.
4. The control loop from the pseudocode above: generate → if action is a tool, execute and append observation → repeat until finish or max steps.

Build order: (1) define tools and a stub env; (2) write one gold trajectory by hand, reasoning out loud; (3) get the loop parsing Thought:/Action: lines reliably; (4) add 2-3 more exemplars covering failure-recovery (a failed search → reformulation); (5) add a max-step cap + CoT fallback.

The 1-2 genuinely hard parts:

• Stopping the loop. The paper’s #1 ReAct-specific failure is repetition. You need a max-step cap, a loop/repetition detector (hash recent action+obs), and ideally a “you seem stuck — try a different approach” nudge thought.
• Parsing reliably. Free-form generation drifts from the format. Use the model’s native function-calling/structured-output instead of regex-parsing Action: lines if you can — far more robust than 2022’s text parsing.

Reach for: LangChain/LlamaIndex ReAct agents (off-the-shelf), or roll your own with native tool-calling APIs (cleaner, fewer surprises). Don’t fine-tune unless you’re optimizing a small model for a narrow, high-volume task — the paper shows fine-tuning helps, but prompting is the 80/20.

How to Improve It

1. Kill the repetition loop with better decoding/control. The authors blame greedy decoding and suggest beam search. Modern alternatives: a working-memory of recent (action, obs) pairs with explicit “don’t repeat” instructions, or a small reflexion step (“am I making progress?”) every K steps. Testable: measure loop-failure rate before/after on ALFWorld.
2. Add explicit self-reflection between attempts (Reflexion). ReAct recovers within a trajectory but doesn’t learn across failed attempts. Bolt on a verbal-reflection memory: on failure, the agent writes a critique that conditions the next attempt. (This became the Reflexion paper — a direct, validated improvement.)
3. Make the action space less artificially weak. The Wikipedia API was crippled on purpose. Swap in a real dense retriever as a tool and let reasoning direct it — you’d likely close much of the gap to supervised SoTA while keeping interpretability.
4. Auto-generate exemplars instead of hand-writing them. Bootstrap (STaR-style): run ReAct, keep trajectories with correct answers, use them as exemplars or fine-tuning data. The paper does this for fine-tuning; do it for prompt-exemplar selection to remove the manual annotation bottleneck.
5. Combine with RL on the trajectory. The authors explicitly flag this. Treat the ReAct trajectory as a policy rollout and reward grounded, non-repetitive, successful traces (think GRPO over agent trajectories) — train the reasoning to be more efficient, not just imitated. This is roughly the direction modern agentic-RL training has gone.
6. Tune dense-vs-sparse thinking adaptively. Right now it’s a fixed per-task choice. A meta-controller that decides “do I need to think before this action?” per step would cut token cost on easy steps and add reasoning on hard ones.

Glossary

• ReAct — Reason + Act: prompting an LLM to interleave reasoning traces (thoughts) with tool actions in one loop.
• Chain-of-Thought (CoT) — prompting that makes the model write intermediate reasoning steps before its answer; reasoning only, no external actions.
• CoT-SC (Self-Consistency) — sample many CoT chains at temperature > 0, return the majority-vote answer; more robust than a single chain.
• Thought / reasoning trace — a language “action” (âₜ ∈ L) that updates the agent’s context but does not touch the environment and returns no observation.
• Action space (A vs Â) — A is the set of real environment actions; Â = A ∪ L adds the unbounded space of language (thoughts).
• Observation (oₜ) — text the environment returns after a real action (e.g., the first 5 sentences of a Wikipedia page).
• Context (cₜ) — the running history of observations, actions, and thoughts fed back to the model each step.
• Few-shot / in-context learning — teaching the model purely via examples placed in the prompt; no weight updates.
• HotpotQA / FEVER — multi-hop question answering / fact-verification benchmarks used for the knowledge tasks.
• ALFWorld / WebShop — long-horizon text-based decision-making benchmarks (household simulation / online shopping).
• BUTLER — an imitation-learning agent baseline for ALFWorld, trained on ~10⁵ expert trajectories per task.
• Imitation Learning (IL) / IL+RL — training a policy to copy expert trajectories, optionally refined with reinforcement learning (reward-driven trial and error).
• Inner Monologue (IM) — prior closed-loop LLM agent whose “monologue” only restates environment state; ReAct’s ablation baseline (ReAct-IM).
• PaLM-540B — the 540-billion-parameter frozen Google LLM used as the base model for the main results.
• Bootstrapping / STaR — generate solutions, keep the correct ones, fine-tune on them to improve a smaller model.
• Hallucination — the model stating fabricated facts as if true; CoT’s dominant failure mode that grounding mitigates.