DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines

modules, then compiles any pipeline of them by simulating it, keeping the runs that pass a metric, and turning those into self-generated few-shot demonstrations (or finetuning data) — no prompt-engineering required.

TL;DR

Every LM pipeline today is built out of hand-tuned prompt strings — long blocks of instructions and examples discovered by trial and error, glued into “chains.” That’s brittle: a prompt tuned for GPT-4 usually breaks on Llama, and a prompt tuned for one task doesn’t transfer to the next. DSPy reframes an LM pipeline as a Python program made of three things: signatures (what a step should do, not how to phrase it), modules (reusable, parameterized implementations of prompting techniques like Chain of Thought or ReAct), and teleprompters (a compiler that automatically generates good demonstrations for every module by running the program and keeping the traces that work). Compiling a few lines of DSPy — with zero hand-written prompts — lifts GPT-3.5 from 24% to up to 88% on GSM8K math problems and lifts a 13B open model to be competitive with hand-prompted GPT-3.5 pipelines on multi-hop QA. It also lets you compile down to a tiny, self-hosted 770M-parameter T5 model that holds its own against proprietary-model pipelines built by hand.

Problem & Motivation

Building an LM pipeline today means writing prompt templates — long strings of instructions, formatting rules, and examples — for every step, for every task, for every model. This is the same move as hand-tuning the weights of a classifier by eyeballing predictions: it works, it’s how everyone did things before automatic optimization existed, and it doesn’t scale. Concretely:

• A prompt string tuned for one LM (say GPT-3.5) often falls apart on another (Llama2), or even on a different subset of the same task, because LMs are notoriously sensitive to exact phrasing.
• Multi-stage pipelines and agents (retrieval → reasoning → answering, or an agent loop with tools) compound this: now several hand-tuned prompts have to work together, and a change anywhere can break the whole chain.
• Popular frameworks like LangChain and LlamaIndex don’t solve this — they package up pre-built chains and tools, but the chains themselves are still implemented with hand-written prompt templates underneath. The paper’s own audit found LangChain’s codebase (Sept 2023) contained 50 strings over 1,000 characters (essentially all hand-written prompts) and 54 files dedicated entirely to prompt templating — versus zero hand-written prompt strings in DSPy.

The pain in one sentence: prompt engineering is manual, per-task, per-model weight-tuning for a system that has no gradient — and nobody has been doing the equivalent of “training” it.

What’s New (Core Contribution)

1. Signatures replace prompt strings. Before: write out "You are a helpful assistant... Question: {q}\nAnswer:" by hand. Now: declare "question -> answer" — a typed description of the transformation — and let the compiler decide how to phrase and support it.
2. Modules replace hand-coded prompting techniques. Before: Chain-of-Thought, ReAct, self-consistency voting, etc. are each implemented as one-off, task-specific prompt templates copied from a paper or repo. Now: ChainOfThought, ReAct, MultiChainComparison, ProgramOfThought are generic, reusable, parameterized modules — drop-in replacements for Predict that work with any signature and any LM, and that carry their own learnable demonstrations (akin to how a nn.Linear layer in PyTorch is generic and learns its own weights).
3. Teleprompters (the compiler) replace manual tuning with optimization. Before: a human decides which few-shot examples go in the prompt. Now: a teleprompter takes the program, a handful of training inputs (labels optional beyond the final output), and a metric — simulates the program, keeps the traces that pass the metric, and turns them into demonstrations or finetuning data for every module in the pipeline, automatically.
4. Prompting and finetuning are unified as one optimization interface. The same bootstrapped traces that build a few-shot prompt for a frozen LM can instead be used as training data to finetune a small model (e.g., T5-Large) for that same pipeline step — and a compiled large-model program can act as a teacher that supervises compiling a cheaper student program.

How It Works (Technically)

DSPy is best understood as “PyTorch, but the trainable parameters are prompts and demonstrations instead of weights, and the LM is the (frozen or finetunable) black box doing the computation.”

Signatures. A signature is just a spec: input_fields -> output_fields, optionally with a task instruction. "question -> answer" means: this module takes a question string and produces an answer string. DSPy expands field names into a natural-language instruction under the hood (e.g. it infers that question and answer play different roles), and — critically — this instruction gets refined by the compiler, not hand-typed by you.

Modules. Predict is the base module. Instantiating dspy.Predict("question -> answer") creates a callable object that stores: (1) the signature, (2) an LM to use (defaults to a global default LM), and (3) a list of demonstrations (starts empty). Calling it does four things: build a prompt from the signature + current demonstrations + the actual input; call the LM; parse the completion back into the output fields; and, if the program is running in “compile mode,” record a trace of (this module, inputs, outputs) to a shared, thread-safe log.

Higher-level modules are just Predict used one or more times under a rewritten signature. The whole implementation of ChainOfThought is: take the user’s signature *inputs -> *outputs, prepend a new output field called rationale (prefixed with “Reasoning: Let’s think step by step.”), and delegate to a single Predict with that expanded signature. ReAct, MultiChainComparison, and ProgramOfThought follow the same pattern — a few lines that reshape the signature and/or call Predict multiple times — which is why the paper can say these “sophisticated” prompting techniques are, underneath, small and fully generic.

Programs. You declare the modules you need in __init__ (so DSPy can find and optimize them later) and wire them together with ordinary Python control flow — loops, ifs, whatever — inside forward(). This “define-by-run” style is lifted directly from PyTorch/Chainer. Here’s the paper’s retrieval-augmented QA example, tracing one input all the way through:

RAG(question="Where is Guaraní spoken?")
 └─ self.retrieve(question)                     # dspy.Retrieve(k=3) -> top-3 Wikipedia passages
      passages = [...three passages about Guaraní...]
 └─ self.generate_answer(context=passages, question=question)   # dspy.ChainOfThought("context, question -> answer")
      1. Predict builds a prompt: instructions for "context, question -> rationale, answer"
         + any demonstrations attached to this module + the actual context/question.
      2. LM generates: rationale="Guaraní is an official language of Paraguay and is also
         spoken in parts of Argentina, Bolivia, and Brazil...", answer="Paraguay (and parts of
         neighboring countries)."
      3. Predict parses the completion into Prediction(rationale=..., answer=...).
 └─ returns Prediction(answer="Paraguay (and parts of neighboring countries).")

Swap the signature to "context, question -> search_query" and the exact same class becomes a query generator instead of an answer generator — nothing else in the code changes. That’s the payoff of separating “what this step does” (signature) from “how it’s phrased” (compiled prompt).

Teleprompters — the compiler. This is the actual novelty. compile(program, trainset, metric) runs in three stages:

1. Candidate generation. Recursively find every distinct Predict module in the program (even ones buried inside ChainOfThought/ReAct/etc.). Run a teacher program (by default, the zero-shot version of the program itself) over the training inputs, with tracing switched on, so every module call is logged as (predictor, inputs, outputs). Feed the final prediction to the user’s metric function. Keep only the full traces that pass — this is rejection sampling: throw away the runs where the pipeline got it wrong, and treat every module call inside a surviving run as a valid demonstration for that module, even though no human ever labeled the intermediate reasoning, search query, or retrieved passage. This is how DSPy gets demonstrations for intermediate steps (like the rationale field or a multi-hop search_query) without you ever writing one.
2. Parameter optimization. Each module now has a pool of candidate demonstrations. Treat “which demonstrations to attach to each module” as a hyperparameter and search over it — random search (BootstrapFewShotWithRandomSearch) or Bayesian/TPE search via Optuna (BootstrapFewShotWithOptuna), scoring each candidate program on a held-out validation set and keeping the best. Alternatively, BootstrapFinetune takes the same bootstrapped traces and uses them as supervised training data to update an actual model’s weights — unifying “pick better few-shot examples” and “finetune a smaller model” under one interface.
3. Higher-order program optimization. The compiler can also rewrite the structure of the program, not just its parameters. The simplest case used in the paper: build an ensemble of several independently-bootstrapped program variants and combine their outputs by majority vote. The paper flags dynamic (test-time) bootstrapping and backtracking as natural next steps here, but doesn’t implement them.

Compiler composition. Because a compiled program is just another program, you can chain compiles: compile a big-model pipeline first, then use that compiled pipeline as the “teacher” whose traces train (via BootstrapFinetune) a small model like flan-t5-large on unlabeled questions — the labels for every intermediate step are bootstrapped, not hand-annotated.

Architecture & data flow

flowchart LR
  SIG["Signature\n'question -> answer'"] --> MOD["Module\nPredict / ChainOfThought / ReAct / ..."]
  MOD --> PROG["Program\nPython class: modules declared in __init__,\nwired by forward()"]
  TRAIN[("Trainset\n+ metric")] --> TP
  PROG -->|compile| TP["Teleprompter\n(the compiler)"]
  TP --> CPROG["Compiled program\n(demonstrations set, or LM finetuned)"]

Stage 1 of compiling, animated: the teacher program runs on several training examples in parallel "lanes." Every module call along the way is logged (the small trace markers). Only lanes whose final output passes the metric (green) get kept — their per-module traces become that module's new demonstrations. Failed lanes (red) are discarded entirely, including their traces.

The algorithm, simplified

This is the core of BootstrapFewShot (the simplest teleprompter), stripped to its essence — it’s literally rejection sampling over multi-step traces:

def bootstrap_fewshot(student, trainset, metric, teacher=None):
    teacher = teacher or student.zero_shot_copy()   # default: simulate the student itself, uncompiled
    compiled = student.deepcopy()

    for example in trainset:
        with tracing_enabled():                     # every Predict.forward() call gets logged
            prediction = teacher(**example.inputs())
            trace = get_recorded_trace()             # [(predictor, inputs, outputs), ...] for this run

        if metric(example, prediction, trace):       # did the FULL pipeline get it right?
            for predictor, inputs, outputs in trace:
                # every module used on this successful run gets a new demonstration,
                # even ones with no ground-truth label (e.g. an intermediate search query)
                demo = Example(**inputs, **outputs)
                compiled.module_named(predictor).demonstrations.append(demo)

    return compiled

The key move: the metric only ever looks at the final output, but a passing run silently certifies every intermediate step that contributed to it — that’s how DSPy bootstraps labels for steps nobody labeled.

Built on Prior Work

Results & Evidence

GSM8K (grade-school math, 200 train / 300 dev / 1.3k test). Three programs (vanilla = direct Predict, CoT = ChainOfThought, reflection = 5-way MultiChainComparison) were each run zero-shot, with random few-shot examples, and compiled with BootstrapFewShot (optionally doubled, optionally ensembled). Headline: across all three programs and both LMs, compiling — not manual prompt-writing — raised accuracy from 4–20% to 49–88%. For GPT-3.5, reflection bootstrap+ensemble hit 86.7% dev accuracy with zero hand-written prompts, beating the human-written Chain-of-Thought variant (78.6%). For llama2-13b-chat, reflection bootstrap+ensemble reached 49.0% dev / 46.9% test — roughly matching or beating the paper’s informal comparison points for much larger models (e.g. llama2-70B’s reported 56.8%, PaLM-540B’s CoT number of 57%) despite using a 13B model with self-generated reasoning chains, not human-written ones.

HotPotQA (open-domain multi-hop QA, ColBERTv2 retrieval). A custom 2-hop retrieve→generate-query→ retrieve→generate-answer program (multihop) was the strongest: GPT-3.5 fewshot 36.9% → bootstrap 48.7% → ensemble 54.7% answer EM on dev. A ReAct tool-use agent compiled with bootstrapping beat its own hand-written-reasoning version (39.0% vs. 33.0% dev EM for GPT-3.5). Compiling also let llama2-13b-chat become “competitive with GPT-3.5” on the same programs. Finetuning case: a 770M-parameter T5-Large, finetuned via BootstrapFinetune with a compiled llama2-13b-chat ensemble as teacher, using only 200 labeled + 800 unlabeled questions, reached 39.3% EM / 46.0% passage accuracy on dev — in the same range as prior published pipelines built on proprietary LMs with hand-written prompts, at a fraction of the inference cost.

GSM8K dev-set accuracy from Table 1, by program and compile strategy, for both LMs. Notice the pattern repeats regardless of program or model: `none`/`fewshot` are weak, `bootstrap` is the big jump, `ensemble` adds a further bump at the cost of more inference calls.

Caveats to keep honest:

• Comparisons to numbers from other papers (Zhang et al. 2022, Wang et al. 2022b, Trivedi et al. 2022, etc.) are explicitly labeled “informal” by the authors — different LMs, eval harnesses, and test samples, so treat them as ballpark, not apples-to-apples.
• Many table cells are blank (–): the authors report dev-set results extensively but only evaluate “promising representatives” on the held-out test set “to avoid overfitting on test” — meaning most of the headline numbers are dev-set, and the test-set subset was chosen after seeing dev results.
• Ensembles (the biggest single jump in several rows) cost 5–7× more inference calls; some of the gain is bought with extra test-time compute, not a free improvement.
• One HotPotQA result is marked as evaluated on only 50% of the test set “due to cost” — a methodological shortcut worth noting if you try to reproduce it.
• The GSM8K metric only checks the final numeric answer, which the authors note let the vanilla program learn to smuggle reasoning into the “answer” field itself — a sign that a loosely specified metric can be gamed in ways that still happen to help, which cuts both ways as a lesson about metric design.
• No confidence intervals are reported for the random-search/Optuna runs; multiple runs are averaged only for the plain few-shot baseline (3–5 runs).

How You’d Use It

This is the most direct “stop hand-prompt-engineering your own pipelines” tool available. Concretely:

• Your harness — model portability. If you want to move a pipeline from GPT-4 to a cheaper or self-hosted model, the DSPy move is: don’t re-write every prompt by hand, re-compile the same program against the new LM with a handful of labeled examples and a metric. That turns a model-migration slog into a repeatable step in your build process instead of a one-off round of prompt-engineering by hand.
• Your business — cost reduction via teacher-student compiling. Compile a pipeline on a strong model first, then use it as the teacher to bootstrap-finetune a small open model (à la the T5-Large result) for production — this is a concrete lever for cutting your own per-call inference cost by orders of magnitude once behavior is validated.
• Your harness — multi-agent / tool-use pipelines. ReAct is a built-in module, and any agent loop you’d build by hand in a multi-agent system is, in DSPy terms, just a module with a signature and a metric over its final action. The teleprompter bootstraps good tool-use traces the same way it bootstraps reasoning chains — useful for tightening up agent reliability without hand-tuning the agent’s system prompt.
• Where it doesn’t replace you: DSPy still needs you to design the signatures and the metric — that’s the actual engineering decision (what should each step’s I/O look like, and what counts as “correct” end to end). It automates the prompt-phrasing labor, not the pipeline design.

Build Your Own (Minimal Recipe)

You can get most of the value with a small, dependency-light implementation:

1. Signature. A tiny parser that turns "question -> answer" into {inputs: ["question"], outputs: ["answer"]}, plus a template function that renders instructions + field prefixes into a prompt string.
2. Predict module. A class holding (signature, lm, demonstrations=[]). Its __call__ renders a prompt from signature + demos + kwargs, calls the LM, and parses the output back into a dict by splitting on the field prefixes.
3. Composable modules. ChainOfThought = Predict with a rationale field prepended to the signature’s outputs — that’s most of the “sophisticated technique” work done. ReAct-style modules just loop Predict calls with a tool-call parsing step in between.
4. Programs. Plain Python classes/functions that call your modules — no special machinery needed beyond keeping a reference to each module instance so you can reach into it later.
5. Tracing. A global (thread-local) list, and a context manager that toggles “recording mode.” Every Predict.__call__, when recording, appends (self, inputs, outputs) to the list.
6. BootstrapFewShot teleprompter. Loop over training examples, run the program with tracing on, check the metric, and on success append each recorded (inputs, outputs) pair as a demonstration on the matching module instance. That’s ~30 lines and captures most of DSPy’s value already (see the pseudocode above).

The genuinely hard parts: (a) getting demonstration selection to generalize from a handful of training examples without overfitting — this is doing model selection with very little data, and it’s where the random-search/Optuna layer earns its keep; (b) writing a metric that’s specific enough to filter out bad traces but loose enough to accept the diversity of correct reasoning/tool-use paths a compiled pipeline will discover — a metric that’s too strict starves the bootstrap of any passing examples to learn from.

How to Improve It

1. Optimize instructions, not just demonstrations. At the time of this paper, teleprompters mostly search over which examples to show; the instruction text and field prefixes are largely fixed by heuristics. Extending the search to instruction wording (later actually pursued in DSPy’s own MIPRO-style optimizers) would close an obvious gap.
2. Smarter candidate selection than random/TPE search. Random search and Optuna are generic; a learned selector — e.g., picking demonstrations by embedding similarity to the validation distribution, or via a small bandit algorithm — could reach the same accuracy with far fewer compile-time trials.
3. RL / self-critique teleprompters. The paper explicitly names this as future work: plug a reward model, self-critique loop (à la Reflexion), or preference signal into the metric so bootstrapping doesn’t require any labels, even for the final output.
4. Dynamic, test-time bootstrapping with backtracking. Right now ensembling is a fixed, compile-time structural change. A runtime controller that detects a likely-failing trace mid-pipeline and backtracks/retries with a different module configuration would push the “higher-order program optimization” stage further, closer to an actual search-time agent.
5. Cost-aware compilation. None of the reported compilers optimize for latency or $/call directly — ensembles buy accuracy with 5–7× more inference. A teleprompter objective that jointly optimizes accuracy-per-dollar (with the teacher→small-model finetuning path as one lever it can pull automatically, not just a manual case study) would make this materially more production-ready for anyone running these pipelines at real volume.

Glossary

• Signature — a declarative spec of a module’s input/output fields (e.g. "question -> answer"), replacing a hand-written prompt string.
• Module — a callable, parameterized component that implements a signature via one or more LM calls (e.g. Predict, ChainOfThought, ReAct); analogous to a PyTorch nn.Module layer.
• Teleprompter — DSPy’s term for a compiler/optimizer: takes a program, training data, and a metric, and returns an optimized version of the program. (“Prompting at a distance,” automated.)
• Compiling — running a teleprompter over a program to bootstrap demonstrations and/or finetune it.
• Demonstration — a concrete (input, output) example attached to a module, used as a few-shot prompt example or as finetuning data.
• Bootstrapping — generating training labels/demonstrations for a pipeline by running it (or a teacher version) and keeping only the outputs that pass a metric.
• Trace — the recorded sequence of (module, inputs, outputs) produced by running a program with tracing/compiling turned on.
• Teacher / student program — in teleprompter composition, the teacher program (often larger or already-compiled) supervises bootstrapping the demonstrations used to compile the student program (often smaller or cheaper).
• Rejection sampling — keeping only the samples (here, full program traces) that satisfy an acceptance criterion (the metric), discarding the rest.
• ReAct — a prompting pattern where the model alternates reasoning steps and tool-call (“action”) steps in a loop.
• Chain of Thought (CoT) — a prompting/module pattern where the model produces an explicit reasoning (“rationale”) field before its final answer.
• Few-shot / in-context learning — giving the model example (input, output) pairs inside the prompt itself, without updating its weights.
• Finetuning — updating a model’s weights on task-specific data, as opposed to prompting a frozen model.
• Ensemble — running multiple compiled program variants and combining their outputs (e.g. by majority vote) into one final answer.
• Exact Match (EM) — a scoring metric that counts an answer correct only if it matches the gold answer exactly (after light normalization).