OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

TL;DR

Most “agent” benchmarks before this either handed the agent a frozen list of clicks to predict (no real environment to act in) or locked it inside one app (just a browser, just a phone screen). OSWorld instead spins up a real Ubuntu/Windows/macOS virtual machine, lets the agent drive raw mouse and keyboard through pyautogui, and checks success by running a script against the final machine state — not by string-matching a “gold” action sequence. The authors hand-built 369 tasks from real forum/tutorial use cases (spreadsheets, image editing, multi-app workflows, even 30 deliberately impossible tasks) with 134 distinct execution-based checkers. When they ran GPT-4V, Gemini, Claude-3, Qwen, and open models against it, the ceiling was 12.24% success versus 72.36% for humans. The bottleneck isn’t planning — models write sensible step-by-step plans — it’s GUI grounding (clicking the right pixel) and operational knowledge (knowing where “brightness” lives in GIMP). This is the benchmark that defined the “computer-use agent” problem and seeded everything from Claude Computer Use to OpenAI’s Operator.

Problem & Motivation

Real digital work spans many apps and both GUIs and command lines: download a file, open LibreOffice Calc, transcribe receipts into a spreadsheet, save it. To measure whether an agent can do that, you need three things prior benchmarks lacked together:

1. A real, executable environment. Datasets like Mind2Web or AITW give you recorded demonstrations and grade by “did you predict the same next action as the human?” That wrongly penalizes any alternative correct path — there are usually ten ways to center a heading. It also forecloses interactive learning: the agent can never try something and see what happens.
2. Open scope across apps and interfaces. WebArena, MiniWoB++, WebShop are real environments, but they’re a browser in a box. They can’t model right-click, Ctrl+click multi-select, dragging between two app windows, or “do this in the terminal then check it in the file manager.” That cap on the action space is a hard ceiling on what you can even ask.
3. Trustworthy, per-task evaluation. General computer tasks don’t reduce to one metric. “Did Amazon’s cookies get deleted?” is a different check than “is column B now sorted?” You need bespoke verification per task, run against the actual machine.

The concrete pain: there was no way to ask “can this model use a computer?” and get an honest, reproducible number. OSWorld exists to give that number.

What’s New (Core Contribution)

• A real-OS, controllable environment (not a simulator). Before: agents were tested in app-specific sandboxes (browser DOM, phone view-hierarchy). Now: a snapshotted VM running actual Ubuntu/Windows/macOS, where the agent emits real pyautogui code (click(300,540), hotkey('ctrl','c')) and the OS executes it. The action space is “everything a human can do with a mouse and keyboard,” not a curated list.
• Execution-based, per-example evaluation at scale. Before: one shared checker per task type, or action-sequence matching. Now: 134 unique verification functions that pull the real artifact (the saved .xlsx, the browser cookie store, the a11y tree) and run a functional correctness check — including dynamic getters (crawlers) for tasks whose “right answer” changes over time (e.g., a paper’s live citation count).
• A 369-task benchmark of genuinely real, multi-app, open-ended work — 27% of it cross-application workflows, 8% deliberately infeasible (deprecated/hallucinated features) to test whether an agent can say “this can’t be done” instead of flailing. 1800+ man-hours of annotation across 9 CS students.
• A baseline study that diagnoses why models fail. Not just a leaderboard: ablations on resolution, history length, window perturbation, and OS transfer that isolate GUI grounding and operational knowledge as the real blockers.

The honest read: the idea of execution-based eval and VM sandboxes existed in pieces. The genuine novelty is putting raw mouse/keyboard control of a real OS together with reliable per-task execution checks at benchmark scale, plus the diagnostic ablations. The benchmark’s influence (it became the de-facto computer-use eval) is the proof it filled a real gap.

How It Works (Technically)

The task as a POMDP

The paper frames each task as a Partially Observable Markov Decision Process (S, O, A, T, R). Don’t let the notation scare you — here’s what each piece actually is:

• S (state): the full state of the computer (every open window, every file). The agent never sees this directly — hence partially observable.
• O (observation): what the agent does see at each step: the natural-language instruction, a screenshot, and/or the accessibility (a11y) tree (a structured XML dump of on-screen UI elements). This is the only window into S.
• A (action): a string of pyautogui Python — click, write('text'), hotkey(...), plus three special tokens WAIT, FAIL (declare infeasible), DONE (declare finished).
• T (transition): what the OS does when you execute the action. The authors don’t model this — the real machine is the transition function. That’s the whole point.
• R (reward): the execution-based checker, returning a value in [0,1] only at the end — 1 for success, a partial decimal for partial credit, 1 for correctly declaring an infeasible task FAIL, else 0.

So the loop is: observe → emit pyautogui code → OS executes → observe again, up to 15 steps, then run the reward script. The agent is given the last 3 observation/action pairs as chat history (more on why below).

The key conceptual move: the environment is real, so the only engineered parts are (a) setting up the initial state and (b) writing the reward checker. Everything in between is just… a computer.

The three things the config file does

Each task is a JSON config with three jobs (color-coded in the paper’s Figure 2):

1. Setup (red): start the VM from a snapshot, download files into it (a half-finished spreadsheet, some receipt images), open the relevant app, resize windows — simulating work already in progress, because real help requests happen mid-task, not from a blank desktop.
2. Post-processing + retrieval (orange/yellow): after the agent stops, save/activate the right window and pull the artifact (file, cookie DB, a11y tree) back to the host for grading.
3. Evaluation (green): call a getter to extract the key component from the final state, then an evaluator (e.g., compare_table(result, expected, rules)) to score it.

This hybrid approach (config + light snapshot, not a full snapshot per task) is deliberate: full per-task VM snapshots would be gigabytes each. The config replays setup steps instead.

Architecture & data flow

flowchart LR
  subgraph Host[Host Machine]
    CFG[Task config JSON]
    COORD[Coordinator]
    TM[Task Manager: setup / postprocess / getters]
    EVAL[Evaluator: 134 exec-based checkers]
    AGENT[LLM/VLM Agent]
  end
  subgraph VM[Virtual Machine - real OS]
    OS[Ubuntu / Windows / macOS]
    APPS[Chrome, LibreOffice, GIMP, VS Code, VLC, terminal]
  end
  CFG --> COORD --> TM --> OS
  OS -- screenshot + a11y tree --> AGENT
  AGENT -- pyautogui action string --> OS
  OS -- final state --> TM --> EVAL
  EVAL -- reward in 0..1 --> COORD

Walking one task through, end to end

Instruction: “Remove the subtitles from this Friends episode and export them as subtitles.srt in the same directory.” (A real success case, GPT-4V.)

1. Setup: config downloads video.mp4 into the VM, opens VLC.
2. Step 1–3: agent screenshots, decides to open a terminal: click(activities), typewrite('terminal'), click(focus).
3. Step 4–5: agent emits two ffmpeg commands — one to extract the subtitle stream to .srt, one to produce a subtitle-free video.
4. Step 6: agent emits DONE.
5. Reward: getter pulls subtitles.srt from the VM; evaluator checks it exists and matches expected content → reward 1.

Notice what made this succeed: the agent escaped the GUI into the CLI, where it has strong training data. The failures (next section) are the GUI-bound tasks where it must click precise pixels.

The four input settings (the real experimental knobs)

Because “what you feed the model” massively changes results, they test four observation formats:

• A11y tree only — the raw XML has millions of tokens, so they filter to visible/interactive elements and flatten to a tab-separated table of (tag, name, text, position, size). The model must output exact pixel coordinates from the position data.
• Screenshot only — raw 1920×1080 image; model must perceive and output precise coordinates. Closest to how humans see; hardest grounding.
• Screenshot + a11y tree — both, hoping the structured text helps the vision model ground.
• Set-of-Mark (SoM) — overlay numbered bounding boxes (drawn from the a11y tree) onto the screenshot, so the model picks an element by index instead of guessing pixels. A known trick that helps web agents — but, surprisingly, often hurts here (too many elements = too much box clutter, and some tasks need sub-element pixel precision).

The algorithm, simplified

The “agent” here is deliberately simple — a prompt loop. The contribution is the environment and checker, so this is what running a baseline actually looks like:

def run_task(task_config, model, max_steps=15, history=3):
    env = OSWorld(task_config)          # boots VM from snapshot, runs setup steps
    obs = env.reset()                   # {instruction, screenshot, a11y_tree}
    trajectory = []                     # rolling (obs, action) memory

    for step in range(max_steps):
        # feed instruction + last `history` (obs, action) pairs as chat turns.
        # screenshots go in as images; a11y tree is filtered to a compact table.
        prompt = build_prompt(obs, trajectory[-history:])
        action = model(prompt)          # -> a pyautogui code string, or DONE/FAIL/WAIT

        if action in ("DONE", "FAIL"):
            break                       # FAIL = "I judge this task impossible"
        obs = env.step(action)          # OS literally executes the pyautogui code
        trajectory.append((obs, action))

    # reward is computed ONLY here, by running the task's bespoke checker:
    #   getter pulls the real artifact (file / cookies / a11y tree) from the VM,
    #   evaluator runs functional correctness -> score in [0, 1]
    return env.evaluate()               # e.g. compare_table(result, gold, rules)

The two hard, hand-built parts are hidden in env.reset() (setup config) and env.evaluate() (the 134 checkers). Everything else is a vanilla ReAct-style loop.

Step through the observe → act → execute loop on a toy task. Each click either lands on the right element (green, progress) or misclicks (red, wasted step) — illustrating how grounding error, not bad planning, eats the 15-step budget. Schematic, not the paper's data.

Built on Prior Work

Results & Evidence

Headline: humans 72.36%, best model (GPT-4 with a11y tree) 12.24%; pure-screenshot best ~5.8%; multi-app workflows top out at 6.57%; some app subsets (LibreOffice Calc) score 0%.

What the ablations actually establish:

• Planning is fine; grounding is broken. Across 550 failed runs, >75% involved mouse-click inaccuracies — the code comments describe the right step, the click lands wrong. This is the single most important finding: the bottleneck is execution/grounding, not reasoning.
• Higher screenshot resolution helps (pure-screenshot setting) — models are trained on images far below 1080p/4k, so they can’t localize precisely on downsampled input.
• More text history helps; more image history doesn’t. Adding past a11y-tree observations boosts SoM performance; adding past screenshots does nothing — VLMs are bad at reasoning over sequences of images. (And a11y trees are huge: ~6,000+ tokens for 90% of single observations, so this gets expensive fast.)
• Agents are brittle to window perturbation. On a subset they solved 50.79% of the time, just moving/shrinking the window or adding clutter dropped success by 60–80%. Humans are unaffected.
• OS transfer is real. Ubuntu→Windows success correlates at 0.7 — methods developed on one OS carry over.
• Variance is huge for models, tiny for humans. Model success swings 0–29% across categories; human performance stays ~70% with <5% variance. Different kind of intelligence.

Caveats to keep honest:

• It’s a 2024 snapshot. The numbers are already obsolete (later computer-use models score far higher) — but the task design and diagnostics are what endure.
• 15-step cap and 3-step history are heuristic; they bound what’s even attempted on long tasks.
• Many ablations run on a 10% subset — directionally useful, not high-precision.
• “Human 72%” is non-expert CS students; an expert would score higher. The human number measures learnable-on-the-fly difficulty, not a hard ceiling.
• False positives/negatives in 134 hand-written checkers are acknowledged and only partly red-teamed.

How You’d Use It

For an AI-services shop, OSWorld is less a thing you ship and more a measurement instrument and design template for the now-hot “computer-use agent” category:

• Honest client evals. When a client asks “can an agent run our back-office workflow in [legacy desktop app]?”, OSWorld’s pattern — VM + raw control + execution-based checker — is exactly how you give a defensible yes/no with a number, instead of a demo that works once. You can fork the harness and author tasks against their apps.
• Regression harness for a computer-use product. If you build an RPA-style agent, OSWorld-style execution checks (verify the final artifact, not the trajectory) are the right CI gate — they tolerate the many valid paths an agent might take.
• Buy-vs-build signal. The 12% ceiling (in 2024) told everyone “general computer-use is not solved — don’t promise full autonomy.” The diagnostic that planning is fine, grounding is the wall tells you where to spend: invest in grounding (good element detection / SoM tuning / accessibility APIs), not in fancier prompt-chains.
• Multi-agent angle. Workflows scoring 6.57% scream for decomposition: a planner agent that delegates per-app subtasks to specialist app-agents, each with its own operational-knowledge prompt — the kind of orchestration you’ve already built. OSWorld is the scoreboard for whether that decomposition actually helps.

Build Your Own (Minimal Recipe)

You can stand up an 80%-of-the-value clone in a weekend:

1. Sandbox. A VM (VirtualBox/QEMU) or a Docker container with a virtual display (Xvfb + a window manager). Snapshot it so you can reset between runs.
2. Eyes + hands. Screenshot via pyautogui.screenshot(); actions by exec-ing the model’s pyautogui string inside the VM. (Sandbox first — you’re literally running model-generated code.) Optionally pull the a11y tree (ATSPI on Linux, pywinauto on Windows).
3. The loop. The run_task function above. Feed instruction + screenshot + last N steps; parse out a pyautogui block; execute; repeat to a step cap.
4. The hard part #1 — the checker. Write a getter (grab the file/state) + evaluator (functional check) per task. This is where the real labor is; start with file-diff and “does X exist” checks before fancy ones.
5. The hard part #2 — initial-state setup. Scripting “download these files, open this app, resize this window” reliably across reboots is fiddly. Budget for it.

Reach-for list: pyautogui (control), Pillow/opencv (screenshots), a VLM with vision (GPT-4o/Claude/Gemini), ATSPI/pywinauto (a11y), and the actual OSWorld repo (it’s open source — fork it rather than rebuild the 134 checkers).

How to Improve It

1. Replace pixel-clicking with a grounding model. Since >75% of failures are click errors, bolt a dedicated GUI-grounding model (e.g., a “describe element → return bbox” detector) between plan and action. Test: does success on Office tasks jump while planning prompts stay fixed?
2. Inject operational knowledge via retrieval. Agents fail in GIMP because they don’t know where “brightness” lives. Give each app a retrieved cheat-sheet of menus/shortcuts (from docs) in context. Cheap to test, likely big lift on Professional-app tasks.
3. Decompose long/multi-app tasks. The 15-step cap + 6.57% workflow score suggest hierarchical planning: a manager that breaks the task into per-app subgoals with sub-budgets. Measure workflow subset specifically.
4. Learn image-history reasoning. The finding that screenshot history doesn’t help is a training gap. Fine-tune (or few-shot harder) on trajectories of screenshots so the agent can use visual memory — then re-run the history-length ablation.
5. Make window-state management a skill. Agents fail when windows move/shrink. Add an explicit “normalize the workspace (maximize target window, close clutter) before acting” sub-policy and re-measure the perturbation-robustness drop.
6. Calibrate the FAIL decision. Some models spam FAIL (false positives on infeasible tasks). A small verifier that double-checks “is this really impossible?” before accepting a FAIL would clean up the infeasible-task scores.

Glossary

• POMDP — Partially Observable Markov Decision Process: a decision problem where the agent sees only partial observations of a hidden true state, picks actions, and gets rewards. Standard frame for RL/agents.
• GUI grounding — translating an intent (“center the heading”) into the exact UI target/pixel to act on. The paper’s #1 failure mode.
• Operational knowledge — knowing how a specific app works (which menu, which shortcut). Distinct from general reasoning.
• a11y tree (accessibility tree) — a structured (XML) representation of on-screen UI elements (tags, names, positions) exposed by the OS for screen readers; here used as a text observation.
• Set-of-Mark (SoM) — prompting trick that overlays numbered boxes on a screenshot so the VLM references elements by index instead of guessing coordinates.
• pyautogui — Python library that scripts mouse/keyboard; OSWorld’s universal action language.
• Execution-based evaluation — scoring by running a check against the real final machine state, rather than matching the agent’s actions to a reference sequence.
• Infeasible task — a task that genuinely can’t be completed (deprecated/hallucinated feature); tests whether the agent will correctly declare FAIL.
• Initial-state setup — scripted preparation that puts the VM into a realistic “work-in-progress” state before the agent starts.
• VLM — Vision-Language Model: an LLM that also takes images (e.g., GPT-4V, Gemini-Pro-Vision, Claude-3 Opus).
• Workflow / multi-app task — a task requiring coordination across several applications (e.g., Chrome + file manager + LibreOffice).