Fundamentals of Building Autonomous LLM Agents

TL;DR

Plain LLMs are stuck as chatbots: no durable memory, no way to act on the world, no ability to re-plan when something breaks. This paper is a structured survey that treats an agent as a mind with four organs — a perception system (turns screenshots, DOM trees, sensor data, or tool outputs into something the LLM can read), a reasoning system (decomposes the task, generates and selects plans, then reflects on failures), a memory system (short-term context plus long-term RAG/SQL/fine-tuned storage), and an execution system (tool calls, code generation, GUI mouse/keyboard actions). The contribution isn’t a new algorithm — it’s a clean taxonomy plus two worked end-to-end examples (a single DPPM+reflection agent and a multi-expert MAS) that tell you exactly which technique to reach for at each stage. The headline real-world number it anchors against: on OSWorld, humans complete ~72% of computer tasks while the best agents hit ~43% — so this is a map of the design space you’d traverse to close that gap, not a claim to have closed it.

Problem & Motivation

A raw LLM is a brilliant amnesiac trapped in a text box. Concretely, it has three structural defects the moment you ask it to do something rather than say something:

1. No persistence. Everything it “knows” about your session lives in the context window and evaporates when the window fills or the call ends.
2. No hands. It cannot click a button, query your database, or call an API unless you build the plumbing.
3. No recovery. When a step fails — a popup appears, a coordinate is off, an element doesn’t exist — a plain model has no mechanism to notice, diagnose, and re-plan.

The paper grounds this in the OSWorld benchmark (agents driving a real operating system to complete open-ended tasks). The authors cite five recurring failure modes that any real GUI agent hits: bad GUI grounding (can’t map a screenshot to the right pixel coordinates), repetitive actions (stuck in a loop), brittleness to window noise (unexpected popups), constrained exploration (e.g. when Set-of-Mark over-prunes the action space), and the blunt ~30-point performance gap vs. humans (43% vs. 72% completion). Prior work tends to attack one organ in isolation — a better vision encoder, a cleverer prompting scheme, a memory trick. What’s missing for a builder is the integration view: which pieces exist, how they connect, and what the smallest competent end-to-end agent looks like. That’s the gap this review fills.

A useful distinction the paper insists on: a workflow is not an agent. If the LLM follows a fixed, designer-authored sequence of steps, you have a workflow — great for predictable tasks, helpless when reality deviates. An agent generates its own strategy from environmental feedback and can re-plan mid-task. Bolting tools and memory onto an LLM doesn’t make it agentic; the closed feedback loop does.

What’s New (Core Contribution)

This is a survey, so “new” means synthesis and framing, not a novel model. The genuine contributions:

• A four-organ decomposition of the agent. Before: agent papers each carve up the system differently, making cross-comparison hard. Now: a consistent perception → reasoning → memory → execution anatomy, with every surveyed technique slotted into exactly one organ. This is the spine of the whole paper.
• A technique catalog per organ, with comparison tables. For each subsystem the authors enumerate the concrete options (e.g. for perception: text-only, multimodal VLM/MM-LLM, structured A11y/HTML trees, tool-augmented) and tabulate strengths/limits/dependencies. This turns “how do agents perceive?” into a decision table you can actually use.
• Two integration walk-throughs. A single-agent reasoning loop built on DPPM (Decompose, Plan in Parallel, Merge) + anticipatory reflection, and a multi-agent version where the same responsibilities are split across specialist “experts” (planning, reflection, error-handling, memory, action, etc.). These are the parts that go beyond summary — they show the wiring.
• A practitioner-oriented framing of build complexity. Stated objective #5 is explicitly to “evaluate the complexity of implementation of each system” — the paper is written for someone deciding what to build, not just what to cite.

Be honest about what’s not new: there are no experiments of the authors’ own, no new benchmark numbers, no released code. The OSWorld figures are quoted from elsewhere. The value is the map, not new territory.

How It Works (Technically)

Think of the agent as one big loop. The environment emits raw signal; perception compresses it into something the LLM can read; reasoning turns the reading plus the goal into a plan and monitors it; memory feeds both perception and reasoning with relevant history; execution turns a chosen step into a real action that changes the environment — and the loop repeats. Let’s walk each organ, then trace one full turn.

Architecture & data flow

flowchart LR
  ENV[Environment\nGUI / web / OS] -->|screenshot, DOM,\nsensor, tool output| PER[Perception System\nVLM + Set-of-Mark +\nA11y/HTML tree]
  PER -->|structured\nobservation| REA[Reasoning System\nDecompose - Plan - Select\n+ Reflection]
  MEM[(Memory System\nshort-term context +\nlong-term RAG/SQL/weights)] <-->|retrieve / write\nexperiences, workflows| REA
  REA -->|chosen step| EXE[Execution System\ntool call / code /\nclick + type]
  EXE -->|action| ENV
  EXE -->|outcome + feedback| REA
  REA -->|store trajectory| MEM

The agent's closed loop. Click "Step" to push one observation→reason→act cycle through the four organs and watch where reflection branches back to re-plan. Schematic, not the paper's data.

1. Perception — turning the world into tokens. The LLM only reads text, so perception’s job is lossy compression of reality into a representation the model can reason over. Four approaches, in rising cost/capability:

• Text-based: the environment already hands you a text description (a chat, a text sim). Perception does nothing — zero overhead, but useless for visual environments.
• Multimodal (VLM / MM-LLM): the heart of GUI agents. A Modality Encoder (a CNN or a Vision Transformer like CLIP/ViT) turns an image into an embedding; an Input Projector maps that visual embedding into the same vector space as the LLM’s text tokens so the backbone can attend over both together; the LLM Backbone reasons; optional Output Projector + Modality Generator let it emit images (e.g. via a diffusion model). The key idea to internalize: a “unified embedding space” means a picture of a button and the word “button” end up as nearby vectors, so attention can relate them. Weakness: VLMs are bad at precise spatial relations and counting.
• Structured-data (A11y tree / HTML): instead of (or alongside) pixels, read the accessibility tree or DOM. This gives you element roles, labels, states (“unread”, “disabled”) and hierarchy for free — semantically precise, no grounding guesswork. OSCAR uses the Windows A11y tree; DualVCR fuses screenshot features with HTML descriptions.
• Tool-augmented: the LLM emits a tool call (web search, weather API, a sensor-reading microservice, a code interpreter), and the result is fed back into context as perception. This is how an agent “perceives” things outside its training cutoff.

A practical enhancer worth knowing: Set-of-Mark (SoM). Rather than asking a VLM “where is the Send button?” (it’ll hallucinate coordinates), you pre-annotate the screenshot with numbered bounding boxes over every interactive element and hand the LLM both the marked image and a list mapping each number to its coordinates + label. Now the LLM picks a number, and your code looks up the exact pixel. This sidesteps the grounding problem and measurably cuts hallucination and miscounting — the paper’s most directly reusable trick.

2. Reasoning — decompose, plan, select, reflect. This is the organ with the most moving parts.

Task decomposition splits a hard task into subtasks. Two families:

• Decomposition-first (HuggingGPT, Plan-and-Solve): break the whole task into sub-goals up front, then plan each. DPPM is the paper’s favored variant — Decompose, Plan in Parallel, Merge: decompose the task, then spin up independent LLM calls/agents to plan each subtask concurrently, then merge the local subplans into one coherent global plan. Parallel planning is the clever bit: because subtasks are planned independently, an error in one subplan doesn’t cascade into the others (the classic failure of sequential planning), and no single agent has to juggle the whole constraint set at once.
• Interleaved (Chain-of-Thought, ReAct): reveal one or two subtasks at a time, adjusting based on live feedback. More fault-tolerant, but long trajectories drift and hallucinate.

Multi-plan generation & selection — because one greedy plan is often wrong, generate several and pick:

• CoT-SC (self-consistency): sample many reasoning paths, take the majority-vote answer. Cheap, no per-step evaluation.
• Tree-of-Thought (ToT) / Graph-of-Thought (GoT): build a tree (or graph) of intermediate “thoughts,” ask the LLM to score each node, and search (BFS/DFS) for the best path. ToT calls the LLM at every step (expensive but deliberate); GoT adds the ability to merge thoughts.
• LLM-MCTS / RAP: use the LLM as the policy/heuristic inside Monte Carlo Tree Search. MCTS is the AlphaGo search: it grows a tree by repeatedly selecting a promising node, expanding it, simulating a rollout to estimate value, and backpropagating that value up the path — balancing exploration of untried branches against exploitation of known-good ones. Here the LLM proposes candidate actions and estimates their value, so you get principled lookahead at the cost of many LLM calls. RAP goes further and has the LLM build a world model to simulate outcomes before committing.

Reflection is the self-improvement loop, and the paper leans on Reflexion (“verbal reinforcement learning”). The trick: instead of updating model weights with a numeric reward (real RL), the agent updates a natural-language memory with a written critique of why it failed, then conditions the next attempt on that critique. Three components: an Actor (LLM that produces a trajectory of actions), an Evaluator (scores the trajectory — exact-match, a heuristic, or another LLM), and a Self-Reflection model (an LLM that reads a sparse success/fail signal + the trajectory and writes specific verbal feedback). It’s “RL” only by analogy — the policy is the prompt, the reward is a sentence, the gradient step is appending text to memory. Anticipatory Reflection (the “Devil’s Advocate” paper) front-loads this: before acting, the agent argues against its own plan, predicts failures, and prepares remedies.

3. Memory — what survives the context window. Two timescales:

• Long-term: survives across sessions. Three implementations — Embodied (bake experience into the weights via fine-tuning; powerful but expensive and impossible on closed models), RAG (embed documents into a vector store; at query time retrieve the top-k relevant chunks and stuff them into context — this is how you ground answers in company files and cut hallucination), and SQL (structured facts queried via text-to-SQL the LLM generates).
• Short-term: the context window itself — a scratchpad for the current task, managed by chunking and summarization.

What to store matters as much as how: experiences (instruction + a trajectory of observation→action pairs, including failed ones, which teach the agent what to avoid), procedures (Agent Workflow Memory induces reusable routines from past successes), knowledge (external facts), and user info (preferences, history — MemoryBank-style). The key management problem is duplication: e.g. collect successful action sequences for a sub-goal in a list, and once it hits 5, have an LLM condense them into one canonical plan.

4. Execution — turning a decision into an effect. Three mechanisms: tool/function calling (LLM emits structured JSON naming a function + params; your runtime executes it), multimodal action spaces (generate coordinate-based mouse/keyboard events to drive any GUI even with no API; or generate-and-run code for data tasks; or motor commands for robots), and the integration challenges that bite in production — latency from chaining vision + action, error propagation across perception/planning/execution layers, and state synchronization keeping the agent’s model of the world consistent.

The algorithm, simplified

Here is the paper’s single-agent example — DPPM with anticipatory reflection — as a loop you could actually type. The novel parts are exposed; model and environment calls are stubbed.

# Single agent: Decompose, Plan-in-Parallel, Merge + reflection loop.
# llm(prompt)->str, embed/retrieve via memory, perceive()->observation, act(step)->outcome

def run_agent(task, memory, max_replans=3):
    subtasks = llm(f"Decompose into independent subtasks:\n{task}")  # decomposition-first

    # Plan each subtask CONCURRENTLY so one bad subplan can't poison the others (DPPM).
    # For each, anticipatory reflection: predict failures and pre-stage remedies.
    subplans = parallel_map(
        lambda s: llm(f"Plan subtask: {s}\n"
                      f"Now play devil's advocate: what could fail? "
                      f"Give a fallback for each risk."),  # Anticipatory Reflection
        subtasks)

    plan = llm(f"Merge these subplans into one coherent plan, "
               f"resolving cross-subtask dependencies:\n{subplans}")  # Merge

    for group in to_executable_groups(plan):       # run in small batches of steps
        obs = perceive()                            # Set-of-Mark'd screenshot + A11y tree
        hints = memory.retrieve(group, obs)         # long-term experience/workflow recall
        outcome = act(llm(f"{group}\nstate:{obs}\nrecall:{hints}"))

        verdict = evaluate(outcome)                 # Evaluator: success / minor / failure
        if verdict == "success":
            memory.store(group, obs, outcome)       # learn the good trajectory
            continue
        if verdict == "minor":
            adjust(group, outcome)                  # nudge coords / retry the step
            continue
        # hard failure -> reflect: is the SUBPLAN wrong, or the whole plan?
        critique = llm(f"Why did this fail?\n{outcome.trace}")   # Reflexion: verbal feedback
        memory.store_failure(group, critique)
        if scope_of(critique) == "subplan" and max_replans:
            plan = replan_subtask(group, critique); max_replans -= 1
        else:
            return run_agent(task, memory, max_replans - 1)       # restart whole plan
    return "done"

The thing to notice: there is no weight update anywhere. All “learning” is text written into memory and re-injected on the next attempt. That’s why this is buildable on top of any closed API model — the intelligence lives in the loop and the prompts, not in fine-tuning.

Built on Prior Work

The paper is a confluence of well-known agent techniques. The lineage it stitches together:

Results & Evidence

This is the section to read skeptically, because the paper runs no experiments of its own. Its evidence is entirely secondary:

• The headline gap (OSWorld): humans ~72.36% task completion vs. leading models ~42.9% (as of June 2025). This is real and sobering, but quoted from the OSWorld leaderboard, not measured here. It establishes that a gap exists, not which of the surveyed techniques close it.
• Perception enhancers help (VCoder, Set-of-Mark): cited experiments show MM-LLMs augmented with these “significantly outperform baseline models on object-level perception… improved counting accuracy and reduced hallucination.” Again, from the original papers — no head-to-head reproduction.
• The integration examples are illustrative, not benchmarked. The DPPM+reflection agent and the multi-expert MAS are described as how you would build it, with no success-rate numbers attached to the specific composition. So you can’t tell from this paper whether DPPM+reflection actually beats plain ReAct on OSWorld.

What the evidence does establish: a coherent, well-cited taxonomy and a credible argument that integrating these four organs is necessary for autonomy. What it does not establish: any ranking of techniques by measured task success, cost, or latency; any ablation; any claim that the recommended composition is optimal. Treat every “improves performance” as a pointer to go read the cited primary source and verify on your own workload. For RQ3 (“how do reasoning strategies affect success/efficiency/cost?”) the paper poses the question well but answers it only qualitatively.

How You’d Use It

For an AI services shop, this paper is a scoping and architecture checklist, not a product. Concrete uses:

• A reference architecture for client agent builds. When a client says “automate our back-office tool that has no API,” you now have a defensible blueprint: Set-of-Mark + A11y-tree perception → DPPM planning → Reflexion-style recovery → tool/GUI execution → RAG memory of past runs. You can quote scope against the four organs.
• A build-vs-buy decision grid. The comparison tables map cleanly onto effort estimates. Text-perception is free; multimodal needs a VLM and preprocessing; structured-data perception needs parsers/automation tooling; tool-augmented needs integration + error handling. Same for memory (RAG is cheap and high-leverage; embodied/fine-tuning is expensive and often impossible on closed models).
• A failure-mode taxonomy for SLAs and demos. The five OSWorld failure modes (grounding, loops, popup noise, over-constrained exploration, the human gap) are exactly the things that blow up in a client demo. Bake guardrails for each into the loop and you differentiate on reliability.
• The “workflow vs. agent” line as a sales qualifier. Many client problems are actually workflows (fixed, predictable) and don’t need an agent at all — cheaper, more reliable to build deterministically. Use the distinction to right-size the engagement instead of over-selling autonomy.

Where it slots in: this is the org chart for the agent runtime you’d assemble on top of LangGraph / a tool-calling API / a browser-automation layer.

Build Your Own (Minimal Recipe)

The smallest agent that captures ~80% of the value — a single-agent GUI/web automator with recovery. Skip the multi-agent MAS until a single agent demonstrably plateaus.

Components & build order:

1. Execution first (the skeleton). Wire tool/function calling on a model that supports it. For GUI tasks add Playwright (web) or an OS automation lib. Get a “click element N / type text” primitive working before anything else — it’s the part that fails silently.
2. Perception via Set-of-Mark + A11y/DOM. For each step, screenshot, run a detector to draw numbered boxes over interactive elements, and dump the DOM/accessibility tree into a {id: {role, label, state, coords}} map. Feed the marked image + map to the LLM and have it choose an id, never raw coordinates. This single choice removes most grounding failures.
3. Reasoning loop. Start with plain ReAct (decompose-as-you-go) — it’s the cheapest thing that works. Add DPPM-style parallel planning only when tasks are long enough that cascading errors hurt.
4. Reflection. Add an Evaluator (start with a heuristic or an LLM-judge) and a Self-Reflection step that, on failure, writes a one-paragraph critique into memory and retries with it. This is the highest-ROI reliability upgrade.
5. Memory. RAG over a vector store of past experiences (instruction + observation/action trajectory, successes and failures). Retrieve relevant trajectories before planning each task.

The genuinely hard parts (budget for these):

• Grounding robustness — even with Set-of-Mark, dynamic UIs, scroll, and popups break the id↔element mapping. You’ll spend real time on re-detection and popup handling.
• Knowing when to re-plan vs. nudge vs. restart — the scope decision in the reflection branch. Too eager → infinite loops; too timid → it gives up. This is tuning, not a one-shot prompt.

Reach for: a strong tool-calling/vision model (frontier API), LangGraph or a hand-rolled state machine for the loop, Playwright for web, a vector DB (e.g. pgvector/Chroma) for experience memory, and an LLM-as-judge for the Evaluator before you invest in anything fancier.

How to Improve It

Limitations the paper names (or implies) that are real leverage points:

1. Actually benchmark the compositions. The paper’s biggest hole is the lack of measured comparisons. Run DPPM+reflection vs. ReAct vs. ToT on OSWorld/WebArena and report success/cost/latency. The first credible composition leaderboard would be genuinely useful and is squarely within the paper’s own RQ3.
2. Cheaper grounding than full SoM. SoM annotates every element each step — expensive and it over-constrains exploration (a stated failure mode). Test incremental/cached marking that only re-detects changed regions, or a hybrid that falls back to raw VLM grounding when the id-map is stale.
3. Reflection that doesn’t just accumulate text. Reflexion’s memory grows unboundedly and can drift. Add a consolidation/forgetting policy (the paper’s own duplication-merging idea, applied to critiques) so reflections stay sharp and the context doesn’t bloat.
4. Learn-from-one-demonstration. The paper floats this as future work and it’s the commercial sweet spot: a human does the task once, the agent induces a reusable workflow (Agent Workflow Memory) and then runs it autonomously. Closing this loop would slash per-client onboarding cost.
5. Anticipatory reflection as a cost lever, not just quality. Front-loading failure prediction could prune bad branches before expensive execution. Measure whether it actually reduces total LLM calls + actions per completed task, not just whether it raises success rate.

Glossary

• Agent vs. workflow — a workflow follows a fixed designer-authored plan; an agent generates and revises its own plan from feedback.
• Perception system — the module that converts environment signals (pixels, DOM, sensors, tool output) into a representation the LLM can read.
• VLM / MM-LLM — Vision-Language Model / Multimodal LLM: models that align images and text in a shared embedding space; MM-LLMs add full reasoning over both.
• Modality Encoder / Input Projector — encoder turns an image into an embedding; projector maps that embedding into the LLM’s text-token space so attention can relate the two.
• Embedding space (unified) — a shared vector space where semantically related image and text items land near each other.
• Set-of-Mark (SoM) — annotating a screenshot with numbered boxes over interactive elements so the LLM picks an id instead of hallucinating coordinates.
• A11y tree — accessibility tree: a hierarchical, labeled representation of UI elements (roles, states) exposed by the OS/browser.
• Task decomposition — splitting a complex task into smaller subtasks; “decomposition-first” plans all up front, “interleaved” reveals subtasks as it goes.
• DPPM — Decompose, Plan in Parallel, Merge: plan subtasks independently/concurrently, then merge, to avoid cascading errors.
• CoT / CoT-SC — Chain-of-Thought reasoning; self-consistency samples many CoT paths and majority-votes the answer.
• ToT / GoT — Tree/Graph-of-Thought: branching reasoning structures whose nodes the LLM scores and searches.
• MCTS — Monte Carlo Tree Search: select→expand→simulate→backpropagate search balancing exploration vs. exploitation; here the LLM supplies the policy/value estimates.
• Reflection (Reflexion) — the agent writes a natural-language critique of its failure into memory and conditions the next attempt on it (“verbal reinforcement learning”).
• Anticipatory reflection — predicting failures and staging fallbacks before acting (the “Devil’s Advocate” pattern).
• RAG — Retrieval-Augmented Generation: retrieve relevant external documents into the prompt to ground answers and cut hallucination.
• Text-to-SQL — having the LLM translate a natural-language question into a SQL query against a structured database.
• Embodied memory — encoding experience directly into model weights via fine-tuning (vs. external memory stores).
• Agent Workflow Memory (AWM) — inducing reusable task routines from past successful trajectories.
• Context window — the maximum tokens an LLM can attend to at once; the agent’s short-term scratchpad.
• Tool / function calling — the LLM emits structured JSON naming a function and arguments, which the runtime executes.
• OSWorld / WebArena / Mind2Web — benchmarks for agents operating real computer/web environments.
• Expert (in MAS) — a specialized agent (planning, reflection, error-handling, memory, action, security…) handling one responsibility in a multi-agent system.