Toolformer: Language Models Can Teach Themselves to Use Tools

TL;DR

Big language models are bad at arithmetic, stale on current events, and shaky in low-resource languages — problems tools like calculators and search engines already solve. The obvious fix, teaching a model to call tools, has always needed either a mountain of human-labeled examples or a hand-built prompt for one specific task. Toolformer removes both requirements: it has the model itself propose API calls inside plain text, executes them, and keeps only the calls that provably reduce the model’s own loss on the following tokens. It fine-tunes on that filtered, self-labeled data and nothing else. The result: a 6.7B GPT-J model that clobbers a same-size baseline and beats the 175B GPT-3 on several benchmarks (LAMA, math word problems), while its plain language-modeling ability doesn’t degrade at all.

Problem & Motivation

Scaling doesn’t fix everything. Even huge LMs hallucinate facts, can’t do reliable arithmetic, don’t know today’s date, and struggle with languages they saw little of in training — and none of that improves much just by adding more parameters. Tools like search engines and calculators already solve these exact problems. But the two ways people had taught models to use tools both had a hard ceiling:

• Heavy human supervision (e.g. WebGPT, LaMDA): humans demonstrate tool use turn by turn. Expensive, and what a human finds useful to look up may not match what the model actually needs.
• Task-specific few-shot prompting (e.g. PAL, MRKL): a hand-crafted prompt with worked examples teaches tool use for one benchmark at a time. It doesn’t generalize — the model never learns a general policy for “when do I need a tool,” it just imitates a template.

Neither path scales to “give the model a toolbox and let it decide for itself, on arbitrary text, when any given tool would help.”

What’s New (Core Contribution)

• Self-supervised tool-use labels, not human ones. Before: someone must write or demonstrate every tool call. Now: the model drafts candidate calls itself via in-context examples (just a handful per tool), and a perplexity-based filter — not a person — decides which calls survive into training data.
• A general, task-agnostic policy. Before: tool use tied to one downstream task via a bespoke prompt. Now: Toolformer is trained on ordinary pretraining text (CCNet) augmented with API calls, so it learns whether and when to reach for a tool as a general skill, tested zero-shot across LAMA, math, QA, multilingual QA, and temporal reasoning.
• A loss-based filter that is the actual training signal. Before: correctness of a tool call was judged by task accuracy or human labels. Now: a call is “useful” purely if inserting it into the text lowers the model’s own next-token loss compared to not calling anything, or calling but withholding the result — an intrinsic, dataset-agnostic criterion.
• No cost to core language modeling. The augmented fine-tuning set contains the exact same underlying text as the original pretraining subset, just with API calls spliced in — so perplexity on held-out text doesn’t move when tool calls are disabled at inference.

How It Works (Technically)

Think of it as three stages applied once to build a training set, followed by ordinary fine-tuning.

1. Sample candidate calls. For a tool like Calculator, you write a short prompt with 2–5 hand-written examples showing text annotated with [Calculator(...)] calls (see Appendix A.2 prompts). Feed a plain text x through the LM prefixed by that prompt. At every token position i, the model gives a probability p_i that an API call token should start there. Keep positions where p_i exceeds a threshold τ_s (top-k at most). At each kept position, sample up to m full candidate calls, e.g. Calculator(400 / 1400).

2. Execute them. Actually run each call — hit the calculator, the Wikipedia BM25 index, the Atlas QA model, the NLLB translator, or the calendar — and get back a text result r_i, e.g. 0.29.

3. Filter by whether the result actually helps. This is the core trick. Define a weighted loss L_i(z) = how surprised the model is by the actual following tokens x_i…x_n when z is prepended as a prefix. Compute three variants:

• L(ε) — no API call inserted at all (baseline).
• L(call, no result) — the call text is there but not the answer.
• L(call, result) — call and answer both inserted, e.g. [Calculator(400/1400) → 0.29].

A call is kept only if L(call, result) is at least τ_f lower than the better of the first two — i.e., seeing the answer must make the rest of the text meaningfully more predictable than either doing nothing or just announcing the call. This single number is Toolformer’s whole notion of “useful”: no downstream task label anywhere in the loop.

4. Fine-tune. Splice the surviving calls into their original texts (x* = x₁:ᵢ₋₁, [call → result], xᵢ:ₙ) and fine-tune the base LM on this augmented corpus with a standard language-modeling loss. Because the underlying text is unchanged, the model doesn’t lose general ability — it just learns, from its own filtered evidence, where reaching for a tool pays off.

Inference is the mirror image of training: decode normally; the moment the model emits the → token (meaning “I’m about to call a tool”), pause generation, actually run the API, insert the real result, then keep decoding. To make the model willing to start calls, decoding checks not just whether <API> is the single most likely next token but whether it’s in the top-k (k=10 worked best) — otherwise a well-calibrated model rarely pulls the trigger on its own.

Architecture & data flow

flowchart TD
  X[Plain text x from CCNet] --> S["Sample candidate positions + calls\n(few-shot prompt per tool)"]
  S --> E["Execute calls\n(Calculator / WikiSearch / QA / MT / Calendar)"]
  E --> F{"Filter: does result reduce\nloss on following tokens by >= tau_f?"}
  F -->|no| DROP[Discard call]
  F -->|yes| SPLICE["Splice call+result into x\nas a prefix segment"]
  SPLICE --> CSTAR["Augmented dataset C*"]
  CSTAR --> FT["Fine-tune base LM (GPT-J)\non C* with LM loss"]
  FT --> TF[Toolformer]

The filtering step for six candidate API calls sampled from real text. Each bar pair shows the loss without any call vs. with the call and its result — the call survives only when the drop clears the threshold τ_f (dashed line). This is the only signal Toolformer ever gets about whether a call is "good."

The algorithm, simplified

# The one loop that generates Toolformer's training data.
# lm(prefix) -> continuation text (sampling); loss(prefix, target_tokens) -> float
def build_dataset_with_tool_calls(texts, tool, prompt, tau_s=0.05, tau_f=1.0, k=5, m=5):
    augmented = []
    for x in texts:                                  # x is one plain-text document
        for i in candidate_positions(x, prompt, tau_s, k):   # where <API> looks plausible
            for call in sample_calls(x, i, prompt, m):        # e.g. Calculator(400/1400)
                result = tool.execute(call)                   # run it for real, e.g. "0.29"
                future_tokens = x[i:]                          # what the model must predict next

                l_none   = loss(x[:i], future_tokens)               # baseline: no call
                l_nores  = loss(x[:i] + f"[{call}]", future_tokens) # call shown, no answer
                l_result = loss(x[:i] + f"[{call} -> {result}]", future_tokens)  # call + answer

                if min(l_none, l_nores) - l_result >= tau_f:   # answer must genuinely help
                    x = insert(x, i, call, result)              # keep it, splice into text
        augmented.append(x)
    return augmented   # fine-tune the base LM on this with plain LM loss

Built on Prior Work

Results & Evidence

Zero-shot, no in-context examples, GPT-J (6.7B) base for all Toolformer variants:

• LAMA (factual cloze): Toolformer +11.7 to +18.6 points over the best GPT-J baseline across SQuAD/Google-RE/T-REx subsets, and beats GPT-3 (175B) too — it calls the QA tool on 98.1% of examples.
• Math (ASDiv, SVAMP, MAWPS): roughly triples accuracy over plain GPT-J (e.g. 9.6% → 40.4% on ASDiv) by calling the calculator on ~98% of examples, beating GPT-3 (175B) as well.
• QA (WebQS, NaturalQuestions, TriviaQA): clear win over GPT-J-scale baselines using Wikipedia search 99.3% of the time, but still trails GPT-3 (175B) — the authors attribute this to a simple, non-interactive BM25 search that can’t reformulate a bad query.
• Multilingual QA (MLQA): translation tool helps for most languages but the picture is muddy — fine-tuning on CCNet itself hurts some languages enough that Toolformer doesn’t beat vanilla GPT-J everywhere, and Hindi barely uses the tool at all (7.3%).
• Temporal reasoning: big wins on a synthetic date-arithmetic set (DATESET, 3.9% → 27.3%) via the calendar tool, but on TempLAMA the calendar is used only 0.2% of the time — the gain there actually comes from search/QA, not the tool the task was designed to showcase. Read: the calendar result generalizes less than it looks.
• Language modeling doesn’t degrade: perplexity on WikiText and held-out CCNet is essentially unchanged with API calls disabled at inference — the core capability didn’t get taxed to buy the tool-use capability.
• Caveats the paper is candid about: it can’t chain tools (output of one feeding another), it can’t use a tool interactively (e.g. refine a bad search query), it’s sample-inefficient (millions of documents → a few thousand useful calculator calls), it ignores the compute/latency cost of calling a tool, and it’s sensitive to exact input phrasing when deciding to call at all. The scaling-law experiment (GPT-2 124M–1.6B plus GPT-J) also shows the whole approach basically doesn’t work below ~775M parameters — small models can’t yet make good use of tools even when given the same training recipe.

How You’d Use It

This is the cleanest recipe available for “give my own model a tool-use reflex without hand-labeling a dataset.” If your application’s LLM keeps hallucinating dates, currency conversions, or lookups that a deterministic function already solves reliably, you don’t have to hand-write a router or a giant few-shot prompt — you can generate a filtered, self-labeled fine-tuning set the same way and bake the tool-calling habit into the model itself. It’s a good complement to (not a replacement for) modern function-calling APIs: those give a model the ability to call a tool when explicitly prompted to consider it; Toolformer’s contribution is training the disposition to reach for it unprompted, mid-generation, on ordinary text — worth the effort if you’re fine-tuning your own smaller model for a narrow application and want that reflex baked in rather than re-prompted on every call. The self-supervised filter is also reusable on its own as an offline data-quality signal for your own harness or pipeline — any time you want to know “did giving the model piece of context X actually help,” the L(without) − L(with) gap is a cheap, task-agnostic way to measure it, useful well beyond tool calls (e.g. deciding which RAG chunks are worth keeping in your own fine-tuning set).

Build Your Own (Minimal Recipe)

You can get most of the value with one tool and a small model. Components:

1. A base LM you can both sample from and fine-tune — anything with an accessible logprob/loss API (open-weight 1B+ model; the paper shows the trick barely works below ~775M).
2. One deterministic tool to start (a calculator is easiest — no retrieval infra needed) plus 3–5 hand-written [Calculator(expr)]-annotated examples as your sampling prompt.
3. A text corpus to mine for candidate positions — doesn’t need to be huge; heuristically pre-filter to documents likely to need the tool (the paper only keeps calculator candidates from texts with ≥3 numbers) to avoid wasting compute on hopeless documents.
4. The filter loop above — this is the one genuinely hard part: computing L(none), L(call, no result), L(call, result) per candidate means running the base model forward three times per candidate at scale, so batch aggressively and cache.
5. Fine-tune on the resulting splice-augmented corpus with plain next-token loss (small learning rate, e.g. 1e-5, short linear warmup).
6. At inference, force <API>-style tokens into the top-k sampling set (don’t require them to be argmax) or the model will rarely volunteer to call anything — this one decoding tweak (k=10 vs. k=1) was the difference between 8.5% and 100% tool usage on WebQS in the paper’s own ablation.

The second-hardest part is picking τ_s and τ_f per tool — too loose and you fine-tune on noise; too tight and you get almost no training examples (the paper’s own calculator yield was ~3,700 out of a much larger scanned CCNet subset).

How to Improve It

• Chain tool calls. Right now every call is generated independently, so “get today’s date, then ask a fact conditioned on that date” is impossible even though the paper explicitly diagnoses this as costing it the TempLAMA win. Sampling multi-call sequences (or letting a call’s result seed the prompt for the next candidate) is a direct, testable fix.
• Make search interactive. Let the model issue a follow-up query or select among multiple returned snippets instead of committing to one BM25 hit — the paper names this as the main reason it loses to GPT-3 on QA.
• Iterate the bootstrap. Apply the same generate→filter→fine-tune loop a second round using the just-fine-tuned Toolformer as the sampler (as STaR-style methods do) — should raise the sample efficiency the paper flags as a weak point.
• Add a cost term to the filter. τ_f currently ignores latency/dollar cost per tool; weighting the loss-reduction threshold by call cost would make the learned policy usable in a production, budget-constrained setting.
• Replace static thresholds with a learned gate. τ_s/τ_f are hand-tuned per tool; a small calibration pass (or the k-based decoding trick generalized) could let the policy learn its own confidence threshold per tool instead of a fixed constant.

Glossary

• API call (in this paper) — a bit of text like [Calculator(400/1400) → 0.29] spliced into a document; the model both writes and later reads these as ordinary tokens.
• Perplexity / loss L(z) — how surprised the model is by real text after being shown prefix z; lower means better predicted. Used here as the sole measure of whether a tool call helped.
• Zero-shot — the model is only given a task instruction, no worked examples, at evaluation time.
• Self-supervised — the training labels (which calls to keep) come from the model’s own loss, not from a human annotator.
• Top-k sampling constraint on decoding — instead of only taking the single most likely next token, allow the <API> token to trigger a call if it’s among the k most likely candidates, making the model more willing to use tools.
• BM25 — a classic keyword-based search-ranking algorithm (not a neural model) used here for the Wikipedia search tool.
• Atlas — a retrieval-augmented LM fine-tuned for question answering, used as the QA tool’s backend.
• NLLB — Meta’s “No Language Left Behind” multilingual translation model, used as the machine-translation tool.
• LAMA / T-REx / Google-RE / SQuAD — benchmarks that test factual recall by having the model fill in a blank in a short statement.
• GPT-J — an open-weight 6.7B-parameter GPT-style model; the base model fine-tuned into Toolformer.