Efficient Agents: Building Effective Agents While Reducing Cost

TL;DR

Modern agent frameworks keep bolting on modules — fancy memory, plan-and-revise loops, Best-of-N sampling, multi-agent committees — and each module quietly multiplies your token bill. This paper does the unglamorous but valuable work of turning each knob one at a time on a hard benchmark (GAIA) and measuring cost-of-pass: the dollars you spend per correct answer, not per attempt. The punchline is counterintuitive for anyone who has been adding complexity hoping for accuracy: the simplest memory beats every elaborate memory scheme, Best-of-N barely moves accuracy while inflating cost, and there’s a sweet spot on planning depth past which you just pay for “overthinking.” They package the winning settings into Efficient Agents, which holds 96.7% of a state-of-the-art open framework’s accuracy while cutting cost 28.4% on the cost-of-pass metric. For anyone shipping agents to paying clients, this is a pricing-and-margin paper disguised as an ablation study.

Problem & Motivation

The concrete pain: agents that work are getting too expensive to run. Products like DeepResearch and Manus are impressive but burn hundreds of LLM calls per task, and a “smarter” agent often just means more calls, longer context, and a bigger reasoning model — all of which compound. If you’re billing a client per task or running a flat-rate product, every redundant module eats your margin directly.

Prior agent research mostly optimizes for the top line: leaderboard accuracy on benchmarks like GAIA. Almost nobody measures the efficiency contribution of each individual module. So practitioners inherit kitchen-sink frameworks (OWL, AutoGen, Smolagents) where planning, memory, tool-use, and test-time scaling are all switched on by default — without evidence that each one earns its cost. The field is at the same inflection point NLP hit after BERT: capabilities first, efficiency later. This paper is the “efficiency later” for agents.

The one-sentence version of the pain: you are paying for agent components that don’t make your agent any more correct.

What’s New (Core Contribution)

This is an empirical/systems paper, so the novelty is in the measurement discipline and the resulting recipe, not a new algorithm.

• First systematic, one-knob-at-a-time efficiency-effectiveness study of agent components. Before: frameworks compared end-to-end on accuracy. Now: a controlled ablation where backbone, planning depth, planning interval, tool config, memory design, and Best-of-N are each varied in isolation against a fixed default, all scored on the same economic metric.
• Cost-of-pass as the agent design metric. Before: accuracy (pass@1) and raw token counts reported separately, so “expensive but accurate” and “cheap but wrong” looked incomparable. Now: a single number — expected dollars per correct answer — that captures the trade-off and even goes to infinity when accuracy is zero (you can never get a right answer, so no price is worth it).
• A set of falsifiable, surprising design rules. “Simple memory beats elaborate memory,” “Best-of-N is mostly wasted spend,” “moderate planning beats maximal planning,” “more search sources + simpler browser ops is both cheaper and better.” These are concrete enough to act on tomorrow.
• Efficient Agents, a config (not an architecture). Before: default OWL-style settings. Now: the lowest-cost-of-pass setting for each component, composed: GPT-4.1 backbone, 8 max steps, plan interval 1, multi-source search, 5 query expansions, no Best-of-N, simple memory. Result: 96.7% of OWL’s accuracy at 28.4% better cost-of-pass.

Be honest about what’s not new: there’s no new model, no new training, no new orchestration algorithm. The contribution is rigor and a recipe. That’s genuinely useful, but it’s a measurement paper.

How It Works (Technically)

The whole paper hangs on one metric and one experimental protocol. Get those and you’ve got it.

The cost-of-pass metric, demystified

The core equation is:

$$v(m, p) = \frac{C_m(p)}{R_m(p)}$$

In plain English: expected dollars to get one correct answer = (cost of a single attempt) ÷ (probability that attempt is correct). If an attempt costs $0.50 and succeeds 50% of the time, you’ll average two attempts to get a right answer, so the cost-of-pass is $1.00. The division by success rate is the whole trick — it punishes a cheap-but-wrong agent and a correct-but-expensive agent on the same scale. When accuracy is 0, you divide by zero → infinity → “no amount of money buys you a correct answer here.”

The cost of a single attempt is just the token bill:

$$C_m(p) = n_{in}(m,p)\cdot c_{in}(m) + n_{out}(m,p)\cdot c_{out}(m)$$

Plain English: (input tokens × input price) + (output tokens × output price). They keep input and output separate because output tokens are far more expensive than input tokens on most APIs — which is exactly why reasoning models that emit thousands of chain-of-thought tokens get punished hard by this metric. $R_m(p)$, the success rate, is just the fraction of correct responses (pass@1).

That’s it. No RL, no gradients, no loss function. The “math” is an accountant’s ratio. The intellectual work is in what they measure it on.

The experimental protocol

They fix a default agent (GPT-4.1, 12 max steps, plan interval 1, simple search, 10 query expansions, no Best-of-N, simple memory) and then vary one component at a time, re-running the full GAIA dev set each time. GAIA is a hard “general AI assistant” benchmark with three difficulty levels (L1 easy → L3 hard) requiring multi-step reasoning, web browsing, and tool use. Holding everything else constant is what lets them attribute a cost or accuracy change to a single knob — the same logic as a controlled lab experiment.

Tracing one knob: memory

Take the most surprising result. They test six memory designs, from Simple (keep only raw observations + actions in context) up to Extra Hybrid (summarize every step into a vector DB, retrieve by cosine similarity, and maintain an LLM-updated long-term memory blob, concatenating all of it each step).

Intuition says richer memory → better reasoning. The data says the opposite: Simple Memory gets 56.36% accuracy at 0.74 cost-of-pass, beating the no-extra-memory baseline (53.33%, 0.98) and every elaborate scheme. Summarized Memory is the worst — 51.52% at 1.52 cost-of-pass — because the summarizer is itself an LLM call that (a) costs tokens and (b) frequently mangles the history, forcing extra attempts. The richer memory schemes add context length (cost) and noise (hurting reasoning) without adding signal. The mechanism: every “smart” module is an extra LLM call, and extra calls are both a cost and a new failure surface.

Why reasoning models lose on this metric

Models like o1 and Claude 3.7 Sonnet get System-2 reasoning via RL-trained long chains of thought — they literally generate thousands of reasoning tokens before answering. (Quick RL primer for context: these models were fine-tuned with reinforcement learning to reward producing reasoning steps that lead to correct answers; the side effect is verbose “thinking” that you pay for as output tokens.) On GAIA, Claude 3.7 is most accurate (61.82%) but its cost-of-pass is 3.54 vs GPT-4.1’s 0.98 — 3.6× worse economics. And it gets worse with difficulty: Claude’s cost-of-pass climbs from 1.69 (L1) to 9.04 (L3), o1 from 1.96 to 12.66. The “overthinking” tax compounds exactly when tasks are hard and you’d most want the model.

Architecture & data flow

flowchart TB
  subgraph Knobs["Components varied one-at-a-time"]
    BB[Backbone LLM]
    PL[Planning: max steps + interval]
    TL[Tool use: search sources, browser ops, query expansion]
    MEM[Memory design]
    TTS[Test-time scaling: Best-of-N]
  end
  Knobs --> AGENT[ReAct-style agent loop]
  AGENT --> GAIA[(GAIA benchmark<br/>L1/L2/L3)]
  GAIA --> ACC[Accuracy pass@1 = R]
  GAIA --> TOK[Token counts in/out]
  TOK --> COST[Cost C = n_in*c_in + n_out*c_out]
  ACC --> COP[cost-of-pass v = C / R]
  COST --> COP
  COP --> PICK{Lowest cost-of-pass<br/>without big accuracy drop?}
  PICK -->|per component| EA[Efficient Agents config]

Interactive: drag the cost and accuracy sliders to feel why cost-of-pass = cost ÷ accuracy punishes both "expensive" and "wrong." Watch the value blow up as accuracy approaches zero.

The memory ablation from Table 5, plotted as accuracy vs cost-of-pass. Simple Memory (bottom-right: high accuracy, low cost) dominates; Summarized Memory (top-left) is the trap. Schematic from the paper's numbers.

The algorithm, simplified

The “algorithm” is the selection procedure that produces Efficient Agents. It’s greedy per-component, not a joint optimization.

# Build an efficient agent by picking, per component, the cheapest
# setting that doesn't tank accuracy. Greedy, one knob at a time.

DEFAULT = dict(backbone="GPT-4.1", max_steps=12, plan_interval=1,
               search="simple", query_expand=10, best_of_n=1, memory="simple")

def cost_of_pass(cfg, bench):
    n_in, n_out, acc = run_agent(cfg, bench)          # full eval on GAIA
    cost = n_in * PRICE_IN[cfg["backbone"]] + n_out * PRICE_OUT[cfg["backbone"]]
    return float("inf") if acc == 0 else cost / acc   # dollars per correct answer

def select_efficient(bench, tolerance=0.03):          # allow ~3% acc give-back
    cfg = dict(DEFAULT)
    base_acc = run_agent(cfg, bench)[2]
    for knob, options in SEARCH_SPACE.items():        # e.g. memory: [simple, summarized, ...]
        best = min(options,
                   key=lambda v: cost_of_pass({**cfg, knob: v}, bench))
        # only accept the cheap choice if accuracy stays within tolerance
        if run_agent({**cfg, knob: best}, bench)[2] >= base_acc - tolerance:
            cfg[knob] = best
    return cfg   # -> GPT-4.1, 8 steps, interval 1, multi-search, 5 expand, no BoN, simple mem

The honest caveat baked into this code: it’s greedy and assumes components are independent. It tunes each knob against the default rather than searching the joint space, so it can miss interactions (a memory setting that’s only good with deeper planning, say). Cheap to run, but not provably optimal.

Built on Prior Work

Results & Evidence

What was tested: Full GAIA dev set, broken into L1/L2/L3, with one-knob ablations on backbone, test-time scaling, planning, tool use, and memory, plus a head-to-head of Efficient Agents vs OWL and Smolagents.

Headline numbers:

• Efficient Agents: cost-of-pass 0.55, accuracy 51.52%, cost $0.228. OWL: cost-of-pass 0.75, accuracy 53.33%, cost $0.398. So Efficient Agents keeps 96.7% of OWL’s accuracy at 28.4% better cost-of-pass and ~43% lower raw dollar cost. Smolagents is the cautionary tale: 53.33% accuracy but cost-of-pass 5.82 — same accuracy, ~10× the cost.
• Memory: Simple beats all (56.36% @ 0.74 cost-of-pass); Summarized is worst (51.52% @ 1.52).
• Best-of-N: N=1→4 pushes tokens 243K→325K but accuracy only 53.33%→53.94%; cost-of-pass rises 0.98→1.28. Money lit on fire.
• Planning: max steps 4→8 jumps accuracy 41.82%→52.73%; 8→12 adds almost nothing but more cost. Eight steps is the knee.
• Tools: more search sources (Google+Wiki+Bing+Baidu+DuckDuckGo) and simpler browser ops are both cheaper and more accurate; more query expansions (3→10) help.
• Backbone dominates everything. It’s the single biggest lever on both accuracy and cost.

What the evidence does NOT establish (read this before quoting the paper to a client):

• One benchmark. Everything is GAIA. GAIA is browsing/research-heavy; the “simple memory wins, Best-of-N is useless” rules may not transfer to coding agents, long-horizon planning, or stateful workflows where memory actually matters.
• Greedy = not optimal. Components tuned independently against a default; interaction effects are unmeasured.
• Tiny absolute drop, narrow margin. 96.7% retention is one config beating another by ~1.8 accuracy points — within run-to-run noise territory for LLM agents, and they don’t report variance/confidence intervals across seeds.
• A typo in the conclusion (“reducing operational cost by xx times”) signals this is an early arXiv v1; treat exact figures as provisional.
• Prices as of May 2025. The entire economic story shifts every time a provider re-prices tokens.

Net read: the direction of every finding is credible and matches production intuition. The exact 28.4% is benchmark- and date-specific. Trust the rules, re-measure the numbers on your own workload.

How You’d Use It

For an AI services company, this paper is a margin playbook, not a research curiosity.

• Instrument cost-of-pass in your own stack. This is the highest-leverage takeaway. Most teams log accuracy and maybe token counts; almost nobody logs dollars-per-correct-answer per task type. Add it and you’ll immediately see which client workflows are quietly unprofitable. This alone is a billable “agent cost audit” offering.
• Default to simple, justify complexity with data. When you stand up a client agent, start with simple memory, ~8 step cap, multi-source search, no Best-of-N. Only add a module when an ablation on that client’s tasks shows it pays for itself. You’ll ship cheaper agents and have evidence for every design choice in the SOW.
• Pick backbones per task tier, not globally. Use a cheap MoE model (Qwen3-30B-A3B class) for L1-style simple tasks and reserve expensive reasoning models for genuinely hard ones. Routing by difficulty is the biggest cost lever the paper surfaces.
• Kill Best-of-N reflexively. If you inherited a framework with BoN/self-consistency on by default, turning it off is often a free ~20-30% cost cut at no accuracy loss. Easy win on day one of a client engagement.
• Reframe the sales conversation. “We can match your current agent’s accuracy at ~40% lower run cost” is a concrete, defensible pitch — and this paper is the citation behind it.

Build Your Own (Minimal Recipe)

You don’t need to reproduce GAIA. The reusable artifact is a cost-of-pass evaluation harness you can point at any agent config.

Components:

1. An eval set with ground truth — 30-100 representative client tasks with known-correct answers. This is the hard, human part; everything else is plumbing.
2. A ReAct-style agent with swappable components (use Smolagents or LangGraph; expose backbone, max_steps, memory mode, tool config as parameters).
3. A token/cost logger — wrap your LLM client to record input/output tokens per call and multiply by current provider prices.
4. The cost-of-pass calculator — literally the 3-line function above.
5. An ablation runner — loop over one knob’s options, run the full eval, record (accuracy, cost-of-pass).

Build order: (1) get the agent running on the eval set and logging tokens → (2) compute cost-of-pass for the default → (3) sweep one knob (start with memory or Best-of-N, the cheap-to-test ones) → (4) greedily lock in the cheapest non-degrading setting → (5) repeat per knob.

The 1-2 genuinely hard parts: building a trustworthy graded eval set (garbage labels → garbage conclusions), and getting deterministic-enough runs to attribute changes to the knob rather than to LLM nondeterminism (run each config 3+ times and average — the paper’s biggest methodological gap).

Reach for: Smolagents or LangGraph for the agent; LangSmith or a simple JSONL logger for traces/tokens; GPT-4.1 or a Qwen3 MoE as the backbone; pandas to crunch the ablation table.

How to Improve It

1. Replace greedy selection with joint search. Components interact (deeper planning may need more memory). Run a small Bayesian optimization or even random search over the joint config space, scored on cost-of-pass — likely finds configs the greedy method misses. Directly testable against their Efficient Agents number.
2. Make it task-adaptive at runtime, not config-time. The paper picks one static config. The real win is a cheap difficulty classifier that routes each incoming task to a backbone+depth tier (cheap MoE + 4 steps for easy, reasoning model + 8 steps for hard). They gesture at “task-adaptive” but ship a static config — close that gap.
3. Add a token budget to the agent loop. Borrow from Token-Budget-Aware Reasoning: estimate a budget per task and have the agent self-terminate or compress when approaching it. Attacks the overthinking tax head-on, especially on the L3 cost blowup.
4. Generalize beyond GAIA. Re-run the same ablation discipline on a coding-agent benchmark (SWE-bench) and a long-horizon planning task. The strong bet is that “simple memory wins” flips where state genuinely matters — and demonstrating where the rules break is itself a paper.
5. Report variance and a Pareto frontier. Multi-seed runs with confidence intervals, plus an explicit accuracy-vs-cost Pareto curve, would turn “trust me, 96.7%” into a defensible claim and let users pick their own point on the trade-off.

Glossary

• GAIA — A benchmark of hard “general AI assistant” tasks (web research, multi-step reasoning, tool use), graded into difficulty Levels 1-3.
• cost-of-pass — Expected dollars to obtain one correct answer: cost-per-attempt ÷ success rate. Goes to infinity when accuracy is 0.
• pass@1 — Accuracy when the agent gets exactly one attempt per task (no retries).
• ReAct — An agent loop that interleaves reasoning (“thoughts”) with actions (tool calls), feeding observations back in.
• Best-of-N (BoN) — Sample N candidate actions/answers, score them with a reward model, keep the best. More samples = more cost.
• PRM (Process/Progress Reward Model) — A model (here, prompted GPT-4o) that scores how promising an intermediate agent step is, used inside Best-of-N.
• Test-time scaling — Spending more inference compute (extra samples/runs) at answer time to boost accuracy, as opposed to training a bigger model.
• MoE (Mixture-of-Experts) — A model that activates only a subset of its parameters per token (e.g. Qwen3-30B-A3B activates ~3B of 30B), giving large-model quality at small-model inference cost.
• System-2 reasoning — Slow, deliberate, multi-step “thinking” (vs fast intuitive System-1), induced in models like o1 via RL-trained long chains of thought.
• Overthinking — When a reasoning model spends excessive tokens on simple problems, inflating cost with no accuracy benefit.
• Chain-of-thought (CoT) — The model writing out intermediate reasoning steps before its final answer; those steps are billed as output tokens.
• Plan interval — How often (every N steps) the agent regenerates its plan from current context, rather than following the original plan blindly.
• Query expansion — Reformulating the user’s search query into several variants to retrieve a broader, more relevant result set.
• Reinforcement learning (RL) — Training a model by rewarding good outcomes; here it’s how reasoning models learned to produce long, answer-improving CoT.