The Scaffolding Matters More Than the Interface

TL;DR

The industry has been repeating a number — “MCP costs 35× more tokens than CLI tools” — that comes from a blog post nobody can reproduce. This team built the controlled experiment: one fixed task (find an issue, branch, patch, commit, open a PR, count files), seven agent scaffoldings, five models, two tool interfaces, every request routed through a proxy that counts tokens, and every run checked against the actual GitHub repository rather than the agent’s own report that it succeeded.

The 35× claim did not survive. Within a single scaffolding, the MCP-to-CLI cost ratio ranged from 0.43× to 29× across thirteen matched pairs, with a median of 0.93× — statistically indistinguishable from nothing, given that re-running an identical configuration already moves the number by 1.5×.

What did survive is bigger and was not what they were looking for. Holding the interface fixed at plain CLI, with no MCP server attached anywhere, the cheapest scaffolding cost 14,660 input tokens per completed task and the most expensive cost 410,797 — a 28× spread produced entirely by the software wrapped around the model. The worst case was a 27-billion-parameter local model that finished the same task for 17,416 tokens under one harness and 2,418,828 tokens under another. Same model, same task, same result: 139× the money.

The one place the two interfaces genuinely separate is the cost of failure. Both arms failed equally often (3 of 19, and 25-of-29 again in a later replication), but 12.9% of the money spent on MCP runs bought nothing, against 2.2% on CLI runs — MCP’s failures landed on the expensive cells. And in a companion experiment where agents had both routes available, only 6 of 21 runs used the MCP tools they were handed. Prompting them to comply did not work; deleting the alternative credential did.

Problem & Motivation

The pain in one sentence: every team shipping agents has to decide whether to expose capability through MCP servers or through plain command-line tools, and the only numbers available to make that call come from three unreproducible blog posts that disagree by an order of magnitude.

The mechanism behind the worry is real and simple. A language model remembers nothing between requests. So every tool schema — the name, the prose description, the parameter list for each operation — has to be re-sent on every single turn. Attach the official GitHub MCP server and you have added 44 tool descriptions to every request the agent makes for the rest of the session. A task that takes twenty exchanges pays for that catalogue twenty times. The CLI approach sends one schema (a shell) no matter how many programs the model eventually invokes, because the knowledge of what gh pr create does is already sitting in the model’s weights.

That reasoning produced the estimates: 35× from MindStudio, lower figures from two other practitioner reports, a range spanning roughly 3× to 35×. All three are unreviewed, all three come from the same community, and — this is the part that matters — each was produced with a different agent scaffolding on a different workload. Nobody held the scaffolding constant. A number obtained by one team with one harness on one job got repeated until it read as a property of the protocol.

There is a second argument in the air, from Peter Steinberger: the shell is represented all over every model’s pre-training corpus (man pages, tutorials, decades of Stack Overflow), while tool-schema syntax arrives mostly in post-training. If that asymmetry is real, small models should struggle with MCP before they struggle with a shell. The paper tests this too, and gets an answer nobody predicted.

So the concrete question a builder actually has: if I wire an MCP server into my own harness instead of a CLI, what does it cost per task, and is that number stable enough to plan around?

What’s New (Core Contribution)

This is a measurement paper. Its contributions are a design and a dataset, not an algorithm.

1. A three-way factorial design that separates scaffolding from interface. Before: every published estimate varied the harness, the task, and the interface at once, so the reported “MCP penalty” was an unknown blend of all three. Now: the task is frozen (same six operations, same repository, same verifier) and three factors vary independently — scaffolding (7), model (5), interface (2). Crucially, two of the seven scaffoldings ship no MCP client at all, giving a control arm that no within-scaffolding comparison can produce. This is the move that turns a debate into a measurement.

2. Verification by repository inspection, not self-report. Before: benchmarks accept the agent’s closing message as evidence of completion. Now: after every run the harness queries the GitHub API and checks four independent conditions — branch exists, file has the patched content, PR was opened, file count reported correctly. An agent that says “Done!” and left no branch scores zero for that condition. In the replication this caught a run that exited cleanly, raised no error, and had completed half the work.

3. Recording which interface the agent actually used, not which it was given. Before: studies assign an interface and assume compliance. Now: the proxy logs every tool call by name. In the companion experiment where both routes were live, 6 of 21 runs used MCP exclusively, 6 used only the shell, 6 mixed, 3 called nothing, and 4 bypassed both and hit the GitHub HTTP API raw. Prompting the agent to use its assigned interface changed nothing material. Any comparison that does not verify this is measuring an unknown mixture — and the bias is directional: an agent that quietly uses its shell while an unused MCP catalogue sits in context produces a suspiciously cheap “MCP” number.

4. A cost-accounting rule most benchmarks get wrong. Cost is computed only over runs that completed the task, because cost-per-unit-of-delivered-work is undefined when nothing was delivered. Failures are then reported separately and in full. This is genuinely novel bookkeeping — and it is load-bearing, because the median failed run in this study burned 170,937 input tokens against 83,600 for the median successful one. Failure is not cheap. An agent that fails works hard, spends double, and hands back nothing.

The honest framing: contributions 1 and 2 are solid methodology anyone can copy. Contribution 4 is the one most likely to change how you read every other agent benchmark you see this year.

How It Works (Technically)

There is no model architecture here. The “mechanism” is the measurement apparatus and the arithmetic of context cost, so that is what to demystify.

The cost arithmetic, in plain terms

Every request an agent sends contains three things: the tool schemas, the accumulated conversation history, and the new material from the last tool result. Only the third is new. So the per-request input is roughly:

input_tokens(turn n) ≈ schemas + history(n-1) + new_output_from_last_tool

Because schemas is a constant re-paid on every turn, the total input over a session is approximately:

total_input ≈ turns × schemas + Σ history growth

Two consequences follow, and the paper leans on both:

• Schema cost is multiplied by turn count, not paid once. Anything that makes an agent take more turns — a confused model, a retry loop, verbose tool output — multiplies the catalogue cost too. This is why the worst cell in the study (Codex + MCP + the 27B model, 108 tool calls, 2.4M tokens) is not a proportional overrun but an order-of-magnitude blowup.
• Caching flattens history but not necessarily schemas across scaffoldings. Providers bill repeated prefix context at ~10% of list price. Cache hit rates in the CLI arm ran 77–98%, so most scaffoldings were caching well. This is why the raw token gap and the dollar gap are different sizes, and why the paper reports both.

The number nobody reports: schemas per request

This is the paper’s sharpest technical observation. The proxy recorded, per request, how many tool descriptions the scaffolding transmitted. Same protocol, same GitHub server, same 44 available operations — and the count varied by a factor of six:

Claude Code’s CLI arm carries 27 built-in tool schemas even with no MCP server attached — roughly 22,000 tokens spent before the task has said a word. That is the single clearest illustration of the paper’s thesis: the “interface tax” people attribute to MCP is often the harness’s own built-in tool list.

The measured consequence: MCP-arm runs delivering 7 schemas per request had a median of 70,836 input tokens; runs delivering all 44 had 216,986. A factor of 3.1 — inside MCP, between two ways of shipping the identical server.

Isolating the arms (why this design detail is the whole experiment)

Naively assigning an interface does not work, because agents cheat. The isolation is physical:

• CLI arm: no MCP server configured at all. Not “instructed not to use” — absent.
• MCP arm: the MCP server holds its own credential; the agent’s shell is pointed at an empty config directory with no credentials, so any fallback to gh fails loudly instead of succeeding silently.

After isolation, all 25 CLI-arm runs with a recorded tool register used only the command line. The MCP arm stayed messier — of 17 runs with a register, 11 used MCP exclusively, 3 of Codex’s mixed in shell commands, one reached for a built-in note tool, and 2 never called an MCP tool at all.

Architecture & data flow

flowchart LR
  ORCH[Orchestration shell script<br/>reset · select cell · invoke · verify]
  subgraph SC[7 agent scaffoldings]
    NOMCP[pi · Tau<br/>no MCP client]
    HASMCP[Claude Code · Codex<br/>qwen-code · Hermes · opencode]
  end
  ORCH -->|subprocess + CLI args| SC
  SC -->|every request| PROXY[LiteLLM proxy<br/>counts input · cached · tool calls · schemas]
  PROXY --> MODELS[(5 models<br/>OpenAI · Anthropic · local GPU)]
  SC -->|CLI arm: gh + git| REPO[(Private GitHub fixture)]
  HASMCP -->|MCP arm: 44 tools| MCPSRV[GitHub MCP server]
  MCPSRV --> REPO
  ORCH --> VER[Verifier<br/>4 conditions via GitHub API]
  VER --> REPO

The proxy is not incidental. Some scaffoldings delegate work to sub-agents, whose token consumption would otherwise go unrecorded and make an expensive harness look cheap. Routing everything through one accounting path closes that hole. (Two Claude Code cells ran on a subscription credential that authenticates only to Anthropic and so bypassed the proxy; on the twelve proxied Claude Code runs, the scaffolding’s self-reported totals matched the proxy to the token, which is why the self-reported cells are trusted.)

The retransmission mechanism. Each bar is one turn of an agent session; the dark band is the tool-schema payload re-sent every single turn. Drag the turn slider to watch the constant become the dominant cost. Schema counts are the paper's measured values (Hermes gateway = 7, opencode MCP = 57, Claude Code MCP = 74); token-per-schema is calibrated from Claude Code's stated ~22k tokens for 27 built-in schemas. Schematic, not the paper's per-run data.

One run, traced end to end

Take the cell codex · qwen3.6:27b · MCP arm — the study’s worst outlier.

1. Orchestrator resets the GitHub fixture to its starting state (so no prior run’s branch can be credited here) and mints a fresh issue number.
2. It invokes codex as an ordinary subprocess with a command line, pointing its model endpoint at the local LiteLLM proxy and its shell at an empty, credential-free config dir.
3. Codex builds request #1: its own tool schemas plus all 44 GitHub MCP tool descriptions plus the task prompt. The proxy logs input tokens, cached share, and schema count.
4. The 27B model emits an MCP tool call. Codex executes it against the MCP server, which uses its own credential to hit GitHub.
5. The tool result is appended to history. Request #2 goes out — carrying the same 44 schemas again, plus the grown history.
6. The model repeats itself. Steps 4–5 run 108 times. Cumulative input: 2,418,828 tokens.
7. The run finishes. The verifier queries the GitHub API: branch exists ✓, file patched ✓, PR open ✓, file count correct ✓ — 4/4, marked complete.
8. Cost is recorded (the run completed, so it counts) at the model’s list rate.

The same model, same task, under Tau’s CLI arm: 7 tool calls, 17,416 tokens, 4/4. Both succeeded. One cost 139× the other. The model was never the constraint; the software around it was.

The algorithm, simplified

The paper’s real contribution as code — the measurement loop and its accounting rule:

SCAFFOLDINGS = ["pi", "tau", "hermes", "codex", "opencode", "qwen-code", "claude-code"]
MODELS       = ["gpt-5.6-luna", "gpt-5.6-terra", "sonnet-5", "glm-5.2", "qwen3.6:27b"]
ARMS         = ["cli", "mcp"]

def run_cell(scaffolding, model, arm):
    reset_fixture()                       # fresh branch state + new issue number, every run

    # Isolation is physical, not instructional. Agents ignore instructions (Section 7).
    if arm == "mcp":
        env = with_mcp_server(GITHUB_MCP, creds="server-only", shell_creds=EMPTY_DIR)
    else:
        env = no_mcp_server_at_all(shell_creds=GH_TOKEN)

    # Every scaffolding is a subprocess with a CLI. That is what makes the study possible at all.
    transcript = subprocess_run([scaffolding, "--non-interactive", "--model-endpoint", PROXY],
                                input=TASK_PROMPT, env=env)

    usage = proxy.usage_for(transcript.session_id)   # input, cached, tool_calls, schemas_per_request

    # Ground truth is the repository, never the agent's closing message.
    done = sum([branch_exists(), file_has_patch(), pr_opened(), count_reported_correctly()]) / 4
    return Run(scaffolding, model, arm, usage, done)


def report(runs):
    completed = [r for r in runs if r.done == 1.0]

    # RULE 1: cost is only defined where work was delivered. Never average a failure into a cost.
    cost = {key: median(r.usage.input for r in group)
            for key, group in group_by(completed, lambda r: (r.scaffolding, r.arm))}
    #        ^ median, not mean: a few runs consumed millions and would swamp an average.

    # RULE 2: where nothing completed, the cell is UNDEFINED — never imputed. Absence is a finding.
    # RULE 3: failure is reported separately, over every run, with nothing excluded.
    wasted = sum(r.usage.input for r in runs if r.done < 1.0)
    return cost, wasted, completion_rate(runs)

Note what step 3 of run_cell implies and the paper says out loud: every scaffolding tested exposes a command-line entry point, which is the only reason you can drive seven of them from one shell loop. There is no way to operate seven different agents through their MCP catalogues from a script. The composability that lets an agent be treated as an ordinary program is the same composability that lets an agent use ordinary programs. They depended on the first in order to measure the second.

Built on Prior Work

Worth noting the intellectual honesty: the authors state that they fixed the task and verification before running the matrix, committed to publishing whatever came out, and then got a result that undercut their own hypothesis. That is rarer than it should be.

Results & Evidence

This is an empirical paper, so this section carries the weight. Read the caveats as carefully as the numbers.

The headline: scaffolding spread at a fixed interface

CLI arm only — no MCP server attached to anything in this table. Median input tokens per completed run.

Two things this establishes cleanly. MCP is not necessary for this class of work — pi and Tau cannot acquire an MCP client, and completed 4/4 each with git, gh and a shell. And the harness spread (28×) dwarfs anything the interface comparison produced. The high cache rates across the whole table (77–98%) rule out “one scaffolding just caches better.”

pi and Tau agree within 12% despite sharing a design philosophy and no code — Tau is an independent Python reimplementation by different authors. That agreement is what lets them attribute the result to the minimal design rather than to one codebase.

The comparison they set out to make: inconclusive

Within each scaffolding that supports both arms, MCP median ÷ CLI median over completed runs:

Restricted to the 13 strictly paired cells (same scaffolding, same model, completed in both arms), ratios span 0.43× to 29.06×, median 0.93×.

Now the number that makes those ratios readable. They re-ran every locally-served configuration three times, changing nothing: median run-to-run spread was 1.51×, widest 5.41×. That is the study’s resolution limit. Nine of the thirteen paired ratios fall between 0.61 and 1.30 — inside the noise band. Four lie outside it, and not all in the same direction: two Codex pairs at 2.27 and 29.06 and one opencode pair at 1.79 favouring CLI, one Claude Code pair at 0.43 favouring MCP.

That bimodal shape is the paper’s explanation for why published estimates disagree by an order of magnitude: each study samples a few points from a two-humped distribution and reports one of them as the value. The quantity is not stable enough to have a value.

Where the arms genuinely separate: the cost of failure

Read this precisely, because it is easy to over-claim. The arms failed the same number of times. The gap in the last column is entirely about which runs failed: the three CLI failures were cheap cells (two runs of the 27B local model, one of the cheapest hosted model) totalling $0.06; the three MCP failures were expensive ones totalling $0.41. A failed MCP run cost about six times a failed CLI run. Six runs cannot tell you whether that is a property of the interface or an accident.

The replication agrees on frequency: 25/29 completed in each arm, weeks later. What survives replication is the cost asymmetry, not a reliability difference.

There is a post-hoc split the authors flag as post-hoc: set aside the smallest model and parity dissolves — CLI failed 1 of 14, MCP failed 3 of 14, wasted-cost shares move to 0.3% vs 18.7%. In the replication, without the smallest model, CLI completed 14/14 and MCP 12/15, with all three failures being the same configuration (qwen-code driving GLM-5.2 with MCP attached), the only cell in the entire study that never completed in either dataset.

The small-model result, and the surprise

Every run of the 27-billion-parameter local model, sorted by cost:

The model completed the task under every scaffolding, and in 10 of the 12 configurations it was given. It was rarely the limiting factor. What varied by 139× was not whether the work got done but what it cost.

And the surprise: both of its shortfalls were CLI-arm cells, and in both cases the same scaffolding completed with MCP attached. Steinberger’s hypothesis predicts the reverse. The authors’ reading, which is the right one: the shell is not the difficulty — under the other five scaffoldings the same model completed 19 of 20 shell-only attempts. The scaffolding sits between the model and either interface, and can make even the familiar one fail.

Cost by model

Token consumption is nearly flat across four of five models (74,755–83,600 median) while cost spans 11× — the variation is price-per-token, not tokens. The open-weight local models were the most reliable in the study.

The study’s own production cost (a result in itself)

Eight days, two Claude Code sessions, 3,610 assistant turns:

Three facts a builder should internalise: the cache carried 97.1% of input; input exceeded output 437:1 (cost models built around output length will misprice agentic work badly); and uncached input was 6,805 tokens across eight days — four ten-thousandths of one per cent.

What this evidence does NOT establish

The authors are unusually good about this, and I’ll add to it.

• The scaffolding comparison is between two populations, not an isolation of one feature. pi and Tau lack MCP clients, but they differ from the other five in every other way too. You cannot attribute their 5–28× advantage to the absence of MCP.
• The delivery-method finding (7 vs 44 schemas, 3.1×) rests on one scaffolding. Four Hermes runs against twelve runs of four other tools. Delivery method is confounded with everything else about Hermes. It is a mechanism worth testing, not one established.
• One task, one domain. GitHub has a mature CLI and an official MCP server. A domain where MCP exposes operations with no convenient CLI equivalent would plausibly favour MCP far more.
• Hosted configurations are largely single-run. Repetitions covered local models only, for budget reasons. One hosted config that happened to be run three times varied by 3.6× — so the resolution limit applies to hosted rows at least as strongly.
• Uneven coverage. Claude Code contributes 6 runs where others contribute 8; its Table-2 row rests on 2 completed runs, one on the study’s dearest model. Hermes’s OpenAI runs did not report tool calls and are excluded from the adherence analysis.
• Three qualifying scaffoldings were not run (gemini-cli, goose, crush) — time, not principle. Cline and OpenHands were excluded structurally: an editor extension and a containerised platform cannot be driven as subprocesses.
• Small n throughout. 54 cells, 46 executed, 40 completed. The 12.9% wasted-cost figure hangs on three runs and the authors say to read it to one significant figure and no further.

One measurement story is worth keeping. Their most expensive MCP cell originally failed because their own config wrote the MCP credential as a variable name the scaffolding only expands in bracketed form. Nothing looked wrong from outside: server attached, catalogue transmitted, MCP tools called, ordinary-looking cost, partial completion, no error anywhere. Only the agent’s closing message named the unexpanded variable. Scored as a failure it put wasted-cost above 40%; re-run with a working credential it completes three times out of three and the figure is 12.9%. “An instrument of this kind does not fail by crashing; it fails by returning a number one would have believed.”

The 54-cell matrix in three dimensions, built from the paper's Appendix A. Each sphere is one run: X = scaffolding, Z = model, Y and colour = log input tokens. Drag to orbit. Note that the height variation runs almost entirely along the scaffolding axis — the arm toggle barely moves a cell, while sliding across scaffoldings moves it two orders of magnitude. Hollow markers are the `void` cells (pi and Tau have no MCP arm to run).

How You’d Use It

You are choosing between MCP servers and plain CLI tools inside your own agent harness. Here is the straight read.

Is MCP’s token overhead real? Partly, and not the way it is being sold.

The 35× number is not defensible. Within a single scaffolding, the median MCP-to-CLI ratio is 0.93× — you cannot tell it from zero. But there is a real, measurable tax, and it is misattributed: it is schemas per request, and it belongs to the calling harness, not the protocol. Same GitHub server, same 44 operations, 7 schemas per request through one client implementation and 74 through another, and a measured 3.1× cost difference between the two delivery styles. When your own agent’s bill looks high, the diagnostic question is not should I use MCP — it is how many schemas is my harness transmitting per request, and do I actually know? Nobody publishes this number by default. Measuring it in your own stack is a cheap, high-payoff afternoon.

When to reach for CLI tools instead, in your own harness:

• The task boundary is known at build time. If your agent’s operations are enumerable when you ship it, a discoverable catalogue describes work it will never do. Wrap the operations in a CLI, put one line in the prompt saying it exists, and let the model’s pre-training carry the syntax.
• The service already has a mature CLI. GitHub, AWS, kubectl, Playwright, Stripe — the operations are already reachable and already in the weights. A playwright MCP server and the playwright CLI expose substantially the same browser operations; only one of them costs you tokens on every turn.
• The workload is high-frequency. Pipeline jobs, cron jobs, any automation running many times a day. The per-task tax compounds; at scale the 5–28× harness spread in this paper is the whole margin on that automation.
• You are running small or self-hosted models. This is the sharpest case for anyone building on local hardware. A 27B model needing 17,416 tokens per task is viable on one workstation. The same model needing 2.4M is not. That difference is a software choice, not a hardware one — and the fix costs you an afternoon of harness work, not a GPU purchase.

When MCP still earns its cost:

• Your agent cannot know in advance which services it will meet. Open-ended assistants, user-configurable integrations, plugin marketplaces. Discoverability is the product.
• The service has no decent CLI. The paper’s scope is explicitly limited to services that do. If you are exposing a proprietary internal API to your own agents, MCP is the standard and building a CLI is extra work for no gain.
• You need a standard integration surface. If the point is “any agent, including ones you didn’t write, can reach this system,” MCP is what the ecosystem already knows how to consume. That is a distribution argument, not a cost argument, and it is a good one — just don’t let it substitute for a cost argument when the caller is your own harness.

What it means for your own builds, concretely:

1. Stop treating MCP overhead as a fixed multiple. It is bimodal. Measure it per-harness, per-workload, in your own stack. That measurement is now a two-day job because the harness is open source.
2. Add “median input tokens per completed task” to your own dashboards. With a fixed task and a proxy, this is a hard number you can track per agent, per release, per model swap — far better than eyeballing a monthly bill.
3. Run a harness audit on your own agents before you scale one. Route your existing agent through a LiteLLM proxy for a week, count schemas per request, count tool calls per completed task, count wasted spend on failed runs. The paper says the spread between harnesses is 5–28×. If you’re on the wrong end of that, fixing it pays for itself in the first month, and you can prove it with your own numbers.
4. Build the verifier before you build the agent. This is the transferable discipline for any automation you hand off to an agent. Every unattended agent should have a small function that checks external state — did the row get written, did the ticket move, did the file land — instead of parsing the agent’s closing message. Agents that believe they succeeded and agents that did succeed produce identical prose.
5. Budget for failure explicitly. Failed runs cost roughly double successful ones and deliver nothing. If you’re forecasting agent cost for a feature or an automation, this is your margin risk, and it is not in most people’s spreadsheets.
6. Do not trust interface assignment. If you configure your agent to use an MCP server and do not remove the alternatives, you are measuring an unknown mixture — and the bias makes MCP look cheaper than it is. Configuration beats instruction, every time, even when it’s your own agent and your own instructions.

Build Your Own (Minimal Recipe)

You can rebuild a version of this scoped to your own stack in two to three days. It is genuinely worth doing, because the answer for your stack will not be the answer in this paper.

Components, in build order:

1. A fixture with a reset. A private repo (or a scratch database, or a sandbox tenant) that a script can restore to a known state. This is the piece people skip and then discover their benchmark has been crediting run N with run N−1’s work.
2. A verifier before an agent. Write the four-to-six independent condition checks first, against the external system’s API. Return a fraction, not a boolean — partial completion is the signal that tells you where things break.
3. A LiteLLM proxy in front of everything. One accounting path. Record per request: input tokens, cached share, tool-call names, and — the one nobody records — schema count. Sub-agents make this non-optional; without it a delegating harness under-reports and looks cheap.
4. A shell orchestrator. for scaffolding in ...; for model in ...; for arm in ... — reset, invoke as a subprocess, capture, verify, append a row. This works only because every candidate harness has a CLI entry point, which is itself part of the lesson.
5. Physical arm isolation. Not prompt instructions. CLI arm: no MCP server configured. MCP arm: server holds its own credential, agent’s shell points at an empty config dir. Verify from the tool-call log that the isolation held.
6. The accounting rule. Cost only over completed runs, median not mean, undefined cells marked undefined rather than imputed, failures reported separately over everything.
7. Repetition, always. Run at least one axis three times to establish your resolution limit before you interpret anything. In this study that limit was 1.51× — which retroactively demoted nine of thirteen headline ratios to noise. Without it you will publish noise as a finding.

The two genuinely hard parts:

• Task design. The authors went through three iterations. Version one (four tool calls) was trivially easy for everything; version two was impossible for most agents; version three was the sweet spot. You need a task hard enough to require multiple turns and easy enough that a mid-tier model finishes it in one session, and every step must leave a durable, checkable trace. Budget real time for this.
• Telling apparatus failures from subject failures. Their credential-expansion bug produced a run that looked entirely normal — server attached, catalogue sent, MCP tools called, plausible cost, partial completion, no error — and would have published their own misconfiguration as a property of MCP. Fix and re-run apparatus failures; score only subject failures; and say in the writeup which kind each one was.

Reach for: LiteLLM (proxy + unified accounting), a private GitHub repo or equivalent sandbox as fixture, the target service’s REST API for verification, plain bash for orchestration, and the authors’ own harness at github.com/Lamb-Project/mcp-vs-cli-bench (Zenodo DOI 10.5281/zenodo.21851992) as a starting point rather than a blank page. For the minimal-harness end of the comparison, read pi and Tau — both are small enough to read in an afternoon, open source, and licensed for modification.

How to Improve It

Five concrete, testable pushes past the paper.

1. Isolate delivery method properly — the highest-value follow-up. The 3.1× gap between 7 schemas and 44 rests on a single scaffolding (Hermes) and is confounded with everything else about it. Fix this by implementing a gateway pair (list_tools / call_tool) as a shim in front of an existing eager scaffolding, so the same codebase runs both delivery styles. That is a genuine A/B, it is a weekend of work, and if it holds it converts the paper’s most interesting observation into an established mechanism — plus a shippable product for anyone running MCP servers today.

2. Add a domain where CLI has no answer. The whole study runs on GitHub, which has gh. Repeat it against a service with an MCP server and no mature CLI (an internal ERP, a niche SaaS). The paper’s scope statement openly invites this. It is the honest test of whether MCP’s cost buys anything, and the result would be genuinely new rather than a replication.

3. Design the study around model scale instead of splitting post-hoc. The most provocative claim — that CLI-arm failures come from transient model capacity while MCP-arm failures are structural and will not be fixed by better models — is offered explicitly as interpretation, not result, and rests on a post-hoc split. Run a proper scale ladder (4B / 8B / 27B / 70B / frontier) × both arms. Cheap on local hardware, since local inference has no marginal cost. If the structural/transient distinction holds it is the most decision-relevant finding in the whole area.

4. Make turn count a controlled variable. Section 4.5 notes that schema cost is multiplied by turns and then leaves it there. Turn count is the multiplier on everything — it explains the 108-call Codex outlier better than the interface does. Instrument tasks of deliberately varying depth (3, 8, 20, 50 expected turns) and fit cost as a·turns·schemas + b·turns + c. That gives you a predictive cost model instead of a table of medians, which is exactly what you need to forecast per-task cost before you ship an automation at scale.

5. Fix the resolution problem with paired-difference statistics on more repetitions. Thirteen paired ratios spanning 70× with a 1.51× noise floor cannot resolve anything. But the paired design is the right design — it just needs n. Local models make repetition free; run each paired cell ten times and report the distribution of log-ratios with a confidence interval instead of a median of thirteen. The finding might well still be “no effect,” but it would be a measured null rather than an inconclusive one, and a measured null is far more useful for killing the 35× myth.

A sixth, lower-effort but commercially sharp: publish schemas-per-request for the top twenty agent harnesses. Nobody reports this number, the paper shows it varies 6× between clients hitting the same server, and collecting it is a scripting job. It would be the most-cited table in the space within a month.

Glossary

• Agent scaffolding (harness) — the software wrapper that takes your request, sends it to the model, executes the tool calls the model asks for, and feeds results back. Claude Code, Codex, opencode are examples.
• Tool schema — a structured description of one operation: its name, a prose explanation of what it does, and its parameter list. Sent to the model so it knows the operation exists.
• MCP (Model Context Protocol) — Anthropic’s standard for how an external service publishes its tool list, so any compatible harness can consume it without custom integration work.
• CLI arm / MCP arm — the two experimental conditions, borrowed from clinical-trial vocabulary. One has an MCP server attached; the other has only a shell and gh.
• Input tokens — the text sent to the model on a request. This is the quantity MCP inflates and the study’s primary cost measure.
• Cached tokens — the portion of input the provider recognises from an earlier request in the same conversation and bills at roughly 10% of list price.
• Schemas per request — how many tool descriptions the harness transmits with each request. Varied 6× between harnesses using the same MCP server; the paper’s sharpest and least-reported metric.
• Eager vs on-demand delivery — whether a harness sends every tool description on every request (eager, e.g. 44 schemas) or sends a two-tool gateway and fetches descriptions only when the model asks (on-demand, 7 schemas).
• Gateway pair — the on-demand pattern: one tool that lists and describes available tools, one that invokes a tool by name. Two schemas standing in for forty-four.
• Completion (4 conditions) — the study’s ground truth, checked against the GitHub API: branch exists, file patched, PR opened, file count reported correctly. Never the agent’s self-report.
• Paired ratio — MCP-arm cost ÷ CLI-arm cost for the same scaffolding running the same model, both completing. The like-for-like comparison; thirteen of these exist.
• Resolution limit — the smallest difference distinguishable from run-to-run noise. Here, 1.51× median spread means anything under roughly 2× is unmeasurable.
• Wasted cost share — the fraction of a group’s spend that went on runs delivering no completed work. 12.9% for the MCP arm, 2.2% for CLI.
• Dense vs mixture-of-experts (MoE) — in a dense model every parameter takes part in every token, so the headline parameter count is also the working memory footprint; MoE activates only a slice, so a 2.4T MoE may be far cheaper to run than the number suggests.
• Counterfactual pricing — the locally-served models cost nothing marginal to run, so the paper prices them at what renting equivalent hosted capacity would cost, to keep one price list across every cell.
• Purposive sample — a sample chosen deliberately for relevance rather than drawn at random; it supports statements about the tools tested, not about all agent harnesses.