TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture

TL;DR

Frontier LLMs reason better when they can call a Code Interpreter and a Search tool, but nobody has published how to combine plain reasoning, code, and search well — and most input questions give no hint about which approach will work. TUMIX answers this by running a heterogeneous mixture of agents (some pure chain-of-thought, some code-only, some search-only, some dual-tool, some “guided”) in parallel, then iterating: each round, every agent re-solves the question while reading all the other agents’ previous answers. An LLM judge decides when refinement has converged (stopping early saves ~51% of compute), and a final majority/LLM vote picks the answer. The result: +3.55% average accuracy over the best prior test-time-scaling method at near-equal cost, and with extra scaling it pushes Gemini-2.5-Pro on Humanity’s Last Exam from 21.6% to 34.1% — past Gemini Deep Research. The non-obvious finding: a diverse group of agents beats repeatedly sampling the single best agent, and you can get an LLM to auto-design even more diverse agents for another +1.2%.

Problem & Motivation

The concrete pain: tool-augmented reasoning works, but it’s a black box. Products like ChatGPT Agent, Gemini-Pro, and Grok4 all claim to use code and search at test time — none publish how. Worse, prior research (Chen et al. 2024b) showed that OpenAI’s Code Interpreter often fails to balance text and code, leaving the coding ability underused: the model reasons in prose when it should have written three lines of Python, or dumps everything into code when commonsense would have nailed it.

The deeper issue is routing under uncertainty. Textual reasoning is great at semantics and commonsense but bad at precise arithmetic and at fetching fresh facts. Code is great at computation. Search is great at knowledge. But a raw question — “what’s the half-life implied by this decay table” vs. “which 2025 paper first proposed X” vs. “prove this inequality” — rarely tells you which mode to use, and the combined text/code/search solution space is huge. Pick wrong and you waste the inference.

Prior test-time-scaling work sidesteps tools entirely. Mixture-of-Agents (MoA) runs many different LLMs and shares answers, but no external tools. Self-MoA then argued diversity doesn’t even matter — just sample the single best LLM repeatedly. So the field had two gaps: (1) no principled way to mix tools, and (2) an open argument about whether agent diversity is worth anything. TUMIX attacks both.

What’s New (Core Contribution)

1. TUMIX itself — a tool-augmented, multi-round, multi-agent test-time scaling framework. Before: MoA-style methods mixed LLMs but not tools; tool methods used a single agent. Now: one LLM drives a fixed pool of 15 heterogeneous tool-use strategies (text / code / search / dual-tool / guided), running in parallel and refining over rounds. +3.55% over the best baseline at matched cost.

2. A diagnosis of why it works — diversity and quality beat brute scale. Before: Self-MoA claimed the best single agent, sampled repeatedly, wins. Now: the paper shows the opposite once tools are in play — a diverse group has higher coverage (probability at least one agent is right) and higher accuracy than 15 samples of the single best agent. Tools (code + search together) specifically increase answer diversity, which is the lever.

3. LLMs as agent designers. Before: agent prompts/frameworks were hand-written by humans. Now: feed the current agents’ code to Gemini-2.5-Pro and ask it to invent more diverse, higher-quality agents; keep the best 15. This auto-design adds +1.2% for free.

4. LLM-as-Judge for adaptive early termination. Before: fixed number of rounds, or stop when votes stabilize. Now: an LLM judges each round whether refinement has converged (with a hard minimum of 2 rounds so it doesn’t quit too eagerly). This holds accuracy while cutting inference cost to ~49% — because over-refinement actively discards correct answers as agents converge on a shared (sometimes wrong) consensus.

How It Works (Technically)

Think of TUMIX as sequential decision-making under a compute budget with a panel of diverse, correlated experts. A question q has an unknown correct answer a*. You have a pool of agents S = {s_1, ..., s_K}. Each agent s_i produces an answer Y_i at cost c_i, and has a competence p_i(q) = P{Y_i = a* | q} — its probability of being right on this question. The whole framework is a policy π deciding, each round: (i) which agents to run, (ii) what each may read from prior rounds (the communication graph), (iii) when to stop, and (iv) how to aggregate into a final answer â_π.

The objective (Eq. 1) is simply:

maximize P{â_π = a*} − λ · Cost_π

In plain English: maximize the chance the final answer is correct, minus a penalty for compute. λ is the knob that trades accuracy for cost; Cost here = total inference calls + input/output tokens. Everything else in the paper is an attempt to push that probability up while keeping cost flat.

Step 1 — The 15 pre-designed agents. These are not different models; they’re one LLM (Gemini-2.5-Pro or -Flash) wrapped in 15 different prompting/tool strategies:

• Base — direct prompt, no scaling.
• CoT — chain-of-thought.
• CoTcode — CoT that outputs code.
• S — search only (the LLM’s inherent web tool).
• C / C+ — Code Interpreter (plain, and a “hinted” version with extra human priors).
• CS — Code Interpreter and Search, with 3 search variants.
• CSG / CSG+ — dual-tool agent steered by a guidance module (CodeSteer), plus an enhanced-prompt version.

Agents with search come in three flavors — Google Search API (gs), the LLM’s inherent search (llm), or both combined (com) — which is how you get to 15. Multi-round tool agents cap tool interactions at 5 per call.

Step 2 — Refinement as message passing. This is the engine. Each round, every agent independently re-solves the question, but its prompt now includes the original question plus all answers from all agents in the previous round. Concretely the round-t prompt to every agent is roughly: "Refine your answer based on {all_answers_previous_round}" concatenated with q. So it’s a fully-connected communication graph — everyone reads everyone.

Two metrics track the dynamics:

• Average accuracy — how often an individual agent is right (quality).
• Coverage (Eq. 2): Coverage(S) = P{ ∪_{i∈S} (Y_i = a*) } — the probability that at least one agent in the set got it right (diversity / exploration). Under independence this is 1 − ∏_i (1 − p_i); with positive correlation between agents it shrinks. Coverage is your ceiling — you can’t select a correct answer that nobody generated.

Here’s the crucial empirical finding (Fig. 3, Fig. 4): coverage decreases monotonically every round. As agents read each other, they converge — partially-correct cases collapse into “all correct” or “all wrong.” Early rounds (1→2) broaden exploration; later rounds homogenize. Average accuracy rises then plateaus (HLE, AIME) or even declines (GPQA). Translation: refinement is a double-edged sword. The first round or two of sharing genuinely helps; after that, the agents talk themselves into a single consensus and throw away minority-but-correct answers.

Step 3 — Optimal stopping. Because of that, you must stop at the right round. Define the expected marginal value of one more round (Eq. 3):

Δ_r = E[A_{r+1} − A_r | signals up to round r]

i.e., how much accuracy you expect the next round to add, given what you’ve seen (vote margin, answer entropy, how fast coverage is collapsing). Stop the first time Δ_r ≤ λ · marginal_cost. In practice TUMIX doesn’t compute this analytically — it queries the LLM (“has this converged enough to finalize?”) with a hard floor of 2 rounds, because LLMs are overconfident and will quit after round 1 if you let them. This Term_LLM strategy hits ~the same peak accuracy as unlimited refinement at 49% of the inferences (and ~46% of tokens, since later rounds are token-heavy).

Step 4 — Final selection. After stopping, take a majority vote across agents, with Gemini-2.5-Pro picking the most consistent output (LLM-as-Selector). Majority vote and LLM-selection both beat picking a random agent — but mostly in early rounds when answers still diverge; once converged, selection barely matters.

Step 5 (the upgrade) — LLM-generated agents. Ask Gemini-2.5-Pro to read the existing agent code and write new agent implementations (prompt + framework, not just prompt tweaks). This yields 25 new agents; keep the best 15 by first-round HLE score. Pool the 30, sample groups of 15, and score each group by a combined diversity+quality metric (Eq. 4):

Combined Score_i = Coverage_i / E[Coverage] + AvgScore_i / E[AvgScore]

This just normalizes coverage and average score to comparable scales and adds them, so you rank candidate agent-groups by both diversity and quality at once. The top groups beat the hand-designed 15.

Architecture & data flow

flowchart TB
  Q[Question q] --> R1
  subgraph R1[Round 1: parallel, independent]
    A1[CoT agent]
    A2[Code / Code+ agents]
    A3[Search agents x3 variants]
    A4[Dual-Tool CS agent]
    A5[Guided CSG / CSG+ agents]
  end
  R1 -->|concat all answers + q| R2
  subgraph R2[Round 2..T: refine on shared answers]
    B1[Each agent re-solves]
    B1 --> B2[Coverage drops, accuracy rises]
  end
  R2 --> J{LLM judge:<br/>converged?<br/>min 2 rounds}
  J -->|no| R2
  J -->|yes| SEL[Majority vote +<br/>LLM-as-Selector]
  SEL --> ANS[Final answer â]

Schematic of the core tension: as refinement rounds increase, coverage (chance someone is right — the ceiling) falls while average accuracy rises then plateaus. The gap is the selection problem. Drag the slider to see why stopping around round 2-3 is optimal. Illustrative, shaped to match the paper's reported dynamics.

The algorithm, simplified

# TUMIX core loop. llm_agent(strategy, prompt) -> answer string.
# judge_converged / select are themselves LLM calls.
def tumix(question, agents, min_rounds=2, max_rounds=7):
    prev_answers = None
    for r in range(max_rounds):
        round_answers = []
        for agent in agents:                       # diverse tool-use strategies
            ctx = question
            if prev_answers is not None:           # message passing:
                ctx += "\nRefine based on:\n" + "\n".join(prev_answers)  # read ALL prior answers
            round_answers.append(llm_agent(agent.strategy, ctx))

        # coverage shrinks each round as agents converge -> must stop in time
        if r + 1 >= min_rounds and judge_converged(question, round_answers):
            break                                  # LLM-as-Judge early termination (~49% cost)
        prev_answers = round_answers

    # selection is mostly a tie-breaker once answers converge; vote anyway
    return select(question, round_answers)         # majority vote + LLM-as-Selector

Built on Prior Work

Results & Evidence

Benchmarks: HLE (2,500 brutal closed-ended questions across many fields — the main testbed), GPQA-Diamond (198 expert PhD-level multiple-choice), AIME 24&25 (60 hard competition math problems). All averaged over 3 runs.

Headline numbers (Table 2), at matched inference cost:

• Gemini-2.5-Pro: TUMIX avg norm 72.3 vs. best baseline (Symbolic-MoE) 70.3 — +2.0%. HLE 21.6→32.3, GPQA 84.6→87.9, AIME 87.3→96.7.
• Gemini-2.5-Flash: TUMIX avg 60.6 vs. best baseline (DEI) 55.5 — +5.9%. The cheaper model gains more.
• TUMIX-Evolve (LLM-designed agents): another step up (72.5 Pro / 62.8 Flash).
• TUMIX+ (extra scaling — repeat inference 4x in first two rounds): HLE 34.1% on Pro, beating Gemini Deep Research’s 26.9% (32.4% with more compute). But efficiency drops hard.

What the evidence establishes well: (1) Tools + diversity genuinely help — the ablations are clean. Going 1→3→15 agents lifts coverage and accuracy; code+search together beats either alone with comparable per-agent quality. (2) Early termination is nearly free accuracy-wise and roughly halves cost. (3) Diversity beats single-best-repeated with tools — a direct, useful rebuttal of Self-MoA in this regime.

Caveats — read these before selling it:

• HLE agent-selection leakage. Several baselines and one TUMIX variant (TUMIX-Evolve) use HLE results to pick agents — marked with * in the table. Those HLE numbers aren’t clean test performance; treat them with suspicion.
• The ceiling is selection, not generation. Coverage on HLE is ≥65% (someone usually gets it right) but accuracy plateaus ~34%. Half the correct answers that exist get thrown away because LLMs can’t reliably pick the right one from noisy candidates. This is the unsolved core problem.
• Cost is real. Test-time scaling needs far more inferences and ~two orders of magnitude more tokens. “Near-equal cost” is only true relative to other scaling methods, not relative to a single call.
• SciMaster underperformed its paper — the authors blame non-open-sourced tools, i.e., results are sensitive to which exact Search/Code backends you wire in. Reproducibility across tool stacks is shaky.
• Three benchmarks, all academic/reasoning. No agentic-workflow, coding-repo, or real-world-task evaluation. Generalization to your client’s messy domain is unproven.

How You’d Use It

This is a test-time orchestration pattern you can ship on frozen models — no training, no fine-tuning, works through any API that exposes code execution and search. For an AI services company, that’s the appealing part: it’s pure prompt/orchestration IP.

• High-stakes Q&A / analysis as a premium tier. Where a wrong answer is expensive (compliance research, technical due diligence, scientific lookup), offer a “deep mode” that runs a TUMIX mixture and an LLM judge. You’re selling accuracy-per-query, and the +3-6% is the moat over a naive single call.
• A drop-in upgrade to an existing single-agent product. You already have a ReAct-style agent with code + search? TUMIX is “run 5-15 strategy variants in parallel, share answers for 2 rounds, vote.” Reuses your existing tool plumbing.
• Cost-controlled scaling. The LLM-as-Judge termination is the commercially important bit: it lets you advertise high accuracy while keeping spend roughly halved versus fixed-round refinement. That turns an expensive feature into a viable SKU.
• The diversity insight reframes your architecture. If you’ve been chasing “the one best prompt,” this says: stop. Heterogeneous strategies with complementary failure modes are worth more than the single strongest one. That’s a cheap, structural win for any multi-agent system you already run.

Realistic effort: a competent engineer can stand up a working prototype in a few days on top of an existing tool-using agent. The hard part is not the loop — it’s the judge/selector quality and managing token spend.

Build Your Own (Minimal Recipe)

The 80/20 version drops auto-designed agents and fancy stopping rules and keeps the engine.

Components:

1. 3-5 diverse agents over one model: CoT (text only), Code (forces Python via Code Interpreter), Search (web tool), Dual-Tool (both), and optionally a CoT-code hybrid. Diversity of strategy is the whole point — don’t just retemperature one prompt.
2. A parallel runner — asyncio.gather over the agents per round. (This is the one Python idiom that matters here: fire all agent calls concurrently, await them all.)
3. The refinement loop — concatenate every agent’s prior answer into the next round’s prompt. Cap at ~3 rounds to start.
4. Selection — majority vote, with an LLM tie-breaker prompt (“here are N candidate answers, pick the most consistent and correct”).

Build order: (a) one agent + tools working; (b) parallelize 3 agents, one round, majority vote — measure coverage vs. single-agent accuracy on a small eval set; (c) add the refinement round and watch coverage drop (you’ll see it); (d) add an LLM judge with a 2-round minimum.

The 1-2 genuinely hard parts:

• Selection is the bottleneck, and it’s hard. Your final accuracy is gated by how well the LLM picks the right candidate. Budget most of your effort here, not on adding more agents.
• Token/cost control. Fully-connected message passing means round-t prompts grow with (#agents × answer length). Truncate/summarize prior answers, or you’ll blow the context window and the budget by round 3.

Reach for: any frontier model with native code execution + web search (Gemini 2.5, GPT-class with Code Interpreter), an async HTTP client, and a sandboxed code executor with a hard timeout (the paper uses 60s, returns “runtime error” for regeneration). A 5-line Agent dataclass holding (strategy_prompt, tools) is enough structure.

How to Improve It

1. Attack the selection bottleneck directly. Coverage ≥65% but accuracy ~34% means a better selector is worth more than any new agent. Try a trained verifier / reward model, pairwise tournament selection, or self-consistency weighted by tool-grounded evidence (e.g., trust answers backed by executed code over prose). This is the highest-leverage research direction the paper itself points to.
2. Stop coverage from collapsing. Refinement homogenizes agents. Inject a diversity-preserving communication graph — don’t show every agent every answer; show partial/contrastive subsets, or add a “devil’s advocate” agent each round whose job is to defend minority answers. Goal: keep coverage flat while accuracy rises.
3. Per-question routing instead of always-run-15. Most of the cost is running all agents on easy questions. Add a cheap difficulty/uncertainty estimate up front and scale the agent count and rounds to it — small panels for easy questions, full mixture for hard ones. Could beat the flat 49% cost reduction substantially.
4. Adaptive agent selection per question, not per benchmark. TUMIX-EvolveD (varying agents per round) underperformed, but that was random. A learned bandit that picks which strategies to deploy based on the question’s features (math → code-heavy, current-events → search-heavy) could raise coverage at lower cost.
5. Confidence-calibrated termination. The paper tried token-confidence and saw no gain, but it queried the LLM naively. A calibrated stopping rule using the actual Δ_r signals it defines (coverage-drop rate + vote margin + entropy) as features for a small learned classifier could beat both Term_LLM and Term_Rule.

Glossary

• Test-time scaling (TTS) — spending more compute at inference (more samples, more agents, more rounds) to get better answers, without retraining the model.
• Coverage — probability that at least one agent in the group produced the correct answer. The hard ceiling on what selection can recover.
• Average accuracy — how often a single agent is right, averaged over agents. Measures quality, not diversity.
• Competence p_i(q) — the probability that agent i answers question q correctly.
• Mixture-of-Agents (MoA) — prior method that runs several different LLMs and aggregates their answers; no external tools.
• Self-MoA — a follow-up claiming the single best agent, sampled repeatedly, beats a diverse mix. TUMIX refutes this in the tool-augmented setting.
• Code Interpreter — a tool that lets the LLM write and execute code (here, sandboxed Python with a 60s timeout) and read the result.
• Refinement / message passing — each round, agents re-answer while reading all agents’ previous answers; a fully-connected communication graph.
• Optimal stopping — deciding which round to halt at; here, an LLM-as-Judge with a 2-round minimum, formalized via the expected marginal gain Δ_r.
• LLM-as-Judge / LLM-as-Selector — using an LLM to decide convergence (Judge) or pick the final answer from candidates (Selector).
• HLE / GPQA / AIME — the three hard reasoning benchmarks: Humanity’s Last Exam (broad), Graduate-level Google-Proof Q&A (science MCQ), American Invitational Mathematics Examination (competition math).
• CodeSteer — a prior steering module that decides when an LLM should use code vs. textual reasoning; reused as TUMIX’s “Guided” agents.
• λ (lambda) — the cost-accuracy trade-off weight in the objective; higher λ means you care more about saving compute.