InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback

TL;DR

Most coding benchmarks score a model on one generation: instruction in, code out, graded, done. That’s not how people code — real programmers run something, read the error, and fix it. InterCode formalizes that loop as a POMDP (agent, environment, reward) and wraps it around Docker containers, so any static text-to-code dataset (Bash commands, SQL queries, Python functions) can become a multi-turn task where the agent sees real execution feedback after every action. The paper builds three such environments from existing datasets (NL2Bash, Spider, MBPP), evaluates 7 models across 4 prompting strategies, and shows the effect is large and consistent: GPT-4 on SQL jumps from 9.1% success in one shot to 73.7% success given 10 turns to interact. It also shows the benefit plateaus — models start repeating themselves and losing the thread after enough turns — which is itself a useful, reusable diagnostic. The framework’s real contribution isn’t the numbers; it’s a ~200-line-per-environment recipe (Dockerfile + dataset + reward function) for turning any coding task into an agent benchmark, which the paper demonstrates by bolting on a fourth task (Capture-the-Flag) almost for free.

Problem & Motivation

Static code-generation benchmarks (HumanEval, APPS, MBPP, CodeXGLUE, Spider) all share the same shape: one instruction goes in, one code block comes out, a grader checks it, the episode is over. That shape has three concrete problems:

• No recovery path. A typo or a wrong assumption in the first (only) generation propagates straight into the score. A human would just run the command, see the error, and fix it — the benchmark never gives the model that chance.
• Disconnected from execution. The generated code is graded by comparing it to reference code or reference output, not by actually running it against the environment it’s meant to affect (a file system, a database). That’s a proxy for the real skill, not the skill itself.
• No standard for interaction. Some prior work does let models see execution results (Jupyter-notebook-style setups, execution-guided synthesis), but each paper invents its own compiler, feedback format, and evaluation procedure. Results across papers aren’t comparable, and every new interactive-coding idea has to rebuild the harness from scratch.

InterCode’s pitch: stop inventing bespoke interactive setups per paper, and instead give the field one general, safe, reusable environment abstraction that any coding dataset can be dropped into.

What’s New (Core Contribution)

• Interactive coding as a standard RL environment, not a bespoke wrapper. Prior “interactive coding” papers (DS-1000, execution-guided synthesis, notebook-style work) each define their own ad-hoc loop. InterCode formalizes the task as a POMDP with an explicit instruction space, action space, observation space, and reward function, and implements it against the OpenAI Gym API — so any RL or LLM-agent tooling built for gym-style environments works here for free.
• Docker as the sandbox, not a scripted sim. Every environment is a real, stateful virtual container (real Bash shell, real MySQL server, real Python interpreter), not a simulated approximation. This is what makes “safe to run genuinely dangerous actions” (rm -rf, sudo, arbitrary SQL) and “reproducible across machines” both true at once.
• A dataset-agnostic construction recipe. Any existing text-to-code dataset can become an InterCode task if it has just two fields — query (instruction) and gold (reference code) — plus a Dockerfile and a reward function. The paper proves this by converting three unrelated datasets (NL2Bash, Spider, MBPP) in under 200 lines of code each, then adding a fourth, structurally different task (CTF) with the same recipe.
• Execution-grounded reward functions that go beyond exact match. Because the reward function has access to the full interaction history and the live container, InterCode can score things exact-match can’t: file-system diffs + content hashes for Bash, order-aware set overlap (Jaccard × Kendall’s τ) for SQL output tables, unit-test pass rate for Python.

How It Works (Technically)

The formal loop. InterCode models the task as a POMDP (U, S, A, O, T, R):

• U — instruction space: natural-language task descriptions (“find all text files and concatenate their names into one file”).
• S — the latent state space: whatever’s true inside the container right now (file contents, DB rows) — the agent never sees this directly.
• A — the action space: any string of code in the target language, or a special submit action.
• O — the observation space: what the container prints back after executing an action (stdout/stderr, or a state diff).
• T: S × A → S — the transition function: executing an action changes the container’s real state.
• R: S × A → [0, 1] — the reward function: scores task completion once submit is issued.

An action is admissible if it parses and executes without error (a syntax error is inadmissible; a wrong but valid command still counts as admissible). Each admissible action advances the container’s real state and returns real output as the next observation. The agent keeps issuing actions — reading feedback, adjusting — until it calls submit, at which point the reward function runs once against the final state and the interaction history. This is mechanically identical to an RL environment’s step() loop; the twist is that the “environment” is a Docker container and the “action” is a snippet of code.

Two metrics come out of this: Success Rate (SR) — the fraction of episodes that score reward = 1 — and Error % — the fraction of issued actions that were inadmissible (a proxy for how often the model is generating code that doesn’t even run, independent of whether it solved the task).

Construction pipeline (how a new InterCode task gets built). Three modular, independent steps:

1. Environment construction — write a Dockerfile that defines the system (Ubuntu + Bash, MySQL, a Python interpreter) and its entrypoint. Docker is chosen specifically because it’s simultaneously safe (a sandboxed container can survive rm -rf / without touching the host), reproducible (the same Dockerfile behaves identically on any machine), and expressive (the Dockerfile DSL can install any real dependency).
2. Data collection — take (or adapt) a dataset where each row has a query (NL instruction) and a gold (reference code/answer). This is the only structural requirement, which is why existing static datasets can usually be reused with light preprocessing rather than rewritten.
3. Reward design — write a function with access to the full action/observation log and the live container, so it can check more than string similarity (e.g., “did the right files actually change, and are their contents correct”).

The three built-in environments (Table 1 in the paper):

NL2Bash in particular needed real rework, because its original instructions are under-specified (“move the file” — which file, where?) and not grounded in any real file system. The authors filtered to ~1000 commands with ≥4 utilities, dropped anything non-Linux/GUI-dependent/using unsupported utilities (ssh, sudo), then hand-grounded 200 of them to specific paths across four purpose-built file systems (one deliberately harder than the others, which is visible later in the results — models consistently score lowest on file system 1).

The InterCodeEnv interface. This is the actual reusable artifact. It’s an abstract class following the OpenAI Gym contract:

• __init__(data_path, image_name, **kwargs) — validates the dataset, boots the Docker container from image_name, sets up logging.
• reset(index=None) — pulls a task instance, resets the container to its initial state, returns the first observation.
• step(action) — logs the (action, observation) pair, executes the action, and if the action is submit, calls get_reward() and save_trajectory(). Returns (observation, reward, done, info) — the standard gym tuple.
• close() — tears down the container cleanly.
• Three methods a subclass must implement per environment: execute_action(action) (how a code string actually gets run against this container type — default is a raw container.exec(action) call with a timeout), get_reward() (task-specific scoring, since “success” means something different per dataset), and reset_container() (how to roll the container back to a clean state between episodes — e.g. git reset --hard; git clean -fd for the Bash file systems).

That’s the entire surface area. Building a new InterCode task is: write a Dockerfile, provide a query/gold dataset, and implement those three methods.

Architecture & data flow

flowchart LR
  U[NL instruction] --> AG[Agent / LLM]
  AG -->|code action or submit| ENV[InterCodeEnv]
  subgraph Container["Docker container (Bash / MySQL / Python)"]
    ST[(Live state:\nfiles, DB, vars)]
  end
  ENV -->|execute_action| Container
  Container -->|stdout / stderr / diff| ENV
  ENV -->|observation| AG
  ENV -->|on submit: get_reward| RW[Reward function]
  RW --> SR[Success Rate + Error %]

The interactive loop (what happens on every turn)

flowchart TD
  RESET[reset: fresh container + first instruction] --> STEP{agent picks an action}
  STEP -->|code| EXEC[execute_action in container]
  EXEC --> OBS[real stdout/stderr becomes observation]
  OBS --> STEP
  STEP -->|submit| REWARD[get_reward against gold + live state]
  REWARD --> DONE[episode ends: SR + Error % logged]

A live InterCode-style episode: each step, the agent emits an action, the container returns real output, and the agent conditions its next action on that feedback — until it submits or runs out of turns. Click "step" to advance.

The algorithm, simplified

This is the core loop every InterCode environment shares — the part a builder would actually reuse (reward functions and Dockerfiles are the task-specific parts you’d swap out):

# Minimal InterCodeEnv-style loop. Real names match the paper's API.
class InterCodeEnv:
    def __init__(self, data_path: str, image_name: str, max_turns: int = 10):
        self.dataset = load_dataset(data_path)         # rows of {query, gold}
        self.container = docker_run(image_name)         # stateful sandbox
        self.max_turns = max_turns

    def reset(self, index=None):
        self.task = self.dataset[index or random_index()]
        self.reset_container()                          # roll state back to clean slate
        self.turn = 0
        return self.task["query"]                        # first observation = the instruction

    def step(self, action: str):
        self.turn += 1
        if action == "submit":
            reward = self.get_reward()                   # task-specific scoring, see below
            done = True
            obs = f"reward={reward}"
        else:
            obs, admissible = self.execute_action(action) # real exec in the container
            reward = 0.0
            done = self.turn >= self.max_turns
        return obs, reward, done, {"admissible": admissible if action != "submit" else True}

    def get_reward(self) -> float:
        # e.g. Bash: lexical_sim(stdout, gold_stdout) * fs_diff_match(self.container, gold)
        raise NotImplementedError  # task-specific: overridden per environment

    def execute_action(self, action: str):
        return self.container.exec(action, timeout=30)   # default: just run it


def try_again_policy(env, llm, max_turns=10):
    """The paper's 'Try Again' prompting strategy: keep feeding execution
    feedback back to the model until it solves the task or runs out of turns."""
    obs = env.reset()
    history = [obs]
    for _ in range(max_turns):
        action = llm(prompt=history)                      # model conditions on ALL prior feedback
        obs, reward, done, info = env.step(action)
        history.append(f"> {action}\n{obs}")               # this is the interaction the paper measures
        if reward == 1.0:
            break
    return reward

Built on Prior Work

Results & Evidence

Interaction helps, a lot, and consistently. Across every model tested (GPT family, PaLM-2 family, Vicuna-13B, StarChat-16B) and every difficulty level/file system, giving the model 10 turns to interact (“Try Again”) beat a single shot. The standout number: GPT-4 on InterCode-SQL goes from 9.1% → 73.7% success rate (all difficulty levels) between Single Turn and Try Again. GPT-3.5-turbo goes from 10.5% → 47.3%. Even the weaker open-source models roughly double their success rate.

The benefit plateaus. Plotting success rate against number of turns used (Figure 3 in the paper) shows most of the gain happens in the first several turns; after that, models increasingly repeat earlier failed actions, ignore recent observations, or keep pursuing a chain of reasoning that’s already gone nowhere. This is worse for harder SQL queries (multi-table joins, several clauses) — models benefit from the ability to interact, but their ability to use a long interaction history well runs out well before the turn budget does.

Schematic of the paper's Figure 3 pattern: success rate climbs fast in early turns, then flattens as late-turn repetition and lost context set in. Toggle a model to see the shape of its curve.

Reasoning-structured prompting beats unstructured retry, with fewer turns. ReAct and Plan & Solve both outperform plain “Try Again” (SQL: 47.3% → 58.7% for ReAct with gpt-3.5-turbo) while using fewer average turns and hitting fewer inadmissible actions — structure helps efficiency, not just final accuracy. But ReAct (flexible, no fixed procedure) generally beats Plan & Solve (rigid plan-then-execute) — Plan & Solve’s upfront plan doesn’t adapt well when execution feedback contradicts it.

Different tasks reward different skills, and no single strategy dominates. SQL success leans on context discovery and error correction (the instructions are phrased as questions; the model learns things about the schema as it goes). Bash success leans more on planning and decomposition (instructions are declarative, multi-step). Consequently ReAct and Plan & Solve only agree on which tasks they solve 57% of the time on SQL and 27.6% on Bash — each strategy is capturing a different, partially non-overlapping slice of capability.

Extensibility is demonstrated, not just claimed. The authors build InterCode-CTF (Capture-the-Flag security puzzles, ~100 instances sourced from picoCTF, spanning categories like forensics/binary exploitation) using the identical Dockerfile + dataset + reward recipe, and show GPT-4 solving a real multi-step forensic-recovery task end-to-end (Figure 4) — evidence the abstraction generalizes past “toy” text-to-code tasks into open-ended, multi-tool problem solving.

Caveats worth holding onto:

• The InterCode-Bash dataset is small (200 instances, adapted from just 1000 filtered NL2Bash commands) and hand-curated/regrounded by the authors — some of that difficulty distribution is an artifact of their curation choices, not an objective ground truth.
• These are prompting-only evaluations. The paper explicitly frames InterCode as an RL environment (state, action, transition, reward all defined) but never actually trains a policy against the reward signal — the RL framing is set up, not exercised. That’s a real limitation, not just a modest claim.
• “Try Again” terminates on reward == 1, which means the model gets to see its own score during the episode for some setups — worth checking exactly how that interacts with the “no external grading during solving” framing before treating SR numbers as directly comparable to single-shot benchmarks.
• The paper’s own limitations section flags the CTF set as still narrow (built by manual curation across a handful of categories) despite the 100-instance headline number — treat CTF results as a proof-of-concept, not a mature benchmark.

How You’d Use It

This maps almost directly onto agentic-coding and computer-use work you’re already doing:

• As the harness pattern for evaluating your own “agent that acts on a real system.” The Dockerfile + query/gold dataset + get_reward() structure is a ready-made template for a regression benchmark on your coding agent, DevOps agent, or database agent — instead of eyeballing transcripts, you get an SR/Error % number you can track across prompt or model changes.
• As the safety pattern for your harness’s execution layer, not just evaluation. The “real, stateful, disposable Docker container the agent acts inside” design is exactly the shape you want for any coding/ops agent in your stack that needs to actually run commands (shell, SQL, Python) without touching production — this is a cleaner reference architecture than ad hoc sandboxing.
• As a due-diligence benchmark when picking a model for your agentic-coding workflow. Before locking your production agent into a specific model/prompting strategy, running an InterCode-style Bash/SQL harness (even a small custom one) tells you concretely how much that model benefits from multi-turn interaction versus how quickly it plateaus and starts repeating itself — directly informs how many turns/retries to budget in production.
• As a component of your own CTF/security-adjacent demo or automation work. The InterCode-CTF pattern (Dockerized vulnerable target + hidden flag + admissible-action logging) is a legitimate, sandboxed way to build agentic-security demos or red-team-style evals for your own defensive research — the same abstraction, pointed at security puzzles instead of Bash utilities.

Build Your Own (Minimal Recipe)

You can get most of the value with a small slice of this:

1. One Dockerfile for the target system (a plain Ubuntu + Bash image is the easiest starting point) with a way to reset state between episodes (a git init + git reset --hard/git clean -fd trick works well for file-system tasks, as the paper itself does).
2. A dataset of {query, gold} pairs — even 20-50 hand-written or LLM-generated instruction/gold-command pairs is enough to start measuring anything.
3. A step()/reset() wrapper around docker exec (use the docker Python SDK) that logs every (action, observation) pair — this is boilerplate, not the hard part.
4. A reward function — start with something crude (exact-match on stdout, or a simple lexical-similarity score) and only add file-diff/hash checks once you see the crude version giving false positives/negatives.
5. A prompting loop — implement “Try Again” first (it’s ~15 lines, shown above); it alone reproduces most of the paper’s headline finding. Add ReAct-style thought/action/observation formatting only once the baseline loop works.

The genuinely hard parts, per the paper’s own experience: (a) reward design — exact match is too strict for almost anything beyond toy tasks, so you need a domain-specific partial-credit function (their Bash reward alone required file-diffs + content hashing + lexical similarity, three separate signals combined), and (b) reliable, fast container reset — if resetting state between episodes is slow or flaky, your eval loop’s iteration speed collapses.

How to Improve It

• Actually train against the reward, not just prompt against it. The paper defines a full RL environment (reward in [0,1], real state transitions) but never trains a policy — an obvious next step is offline RL or verbal-reinforcement-style training (à la Reflexion) directly on the InterCode reward signal instead of only running frozen models with different prompts.
• Replace the fixed turn budget with adaptive stopping. Since success rate plateaus and error behavior (repetition, ignoring feedback) increases in late turns, a policy that estimates “is more interaction still helping?” and stops or resets its context could recover a chunk of the wasted late-turn budget — a small addition on top of the existing loop.
• Add a memory/retrieval layer across turns. The paper notes models lose track of relevant history as the transcript grows; instead of stuffing the full raw trajectory into context, retrieving only the relevant prior observations (a standard MAS memory pattern) is a directly testable fix, and InterCode’s logged trajectories are exactly the data you’d need to evaluate it.
• Combine ReAct and Plan & Solve instead of choosing one. The paper shows the two strategies solve largely non-overlapping subsets of tasks (57%/27.6% overlap) — a hierarchical agent that plans first (Plan & Solve) then executes each step with ReAct-style adaptive re-planning is a concrete, testable hybrid the paper’s own data motivates.
• Grow InterCode-CTF into a real benchmark. The 100-instance CTF set is explicitly a proof of concept; scripting Dockerfile + flag generation for picoCTF-style challenges at scale (rather than manual curation) would turn “we can bolt on a new task type” into a genuinely useful agentic-security benchmark.

Glossary

• POMDP — Partially Observable Markov Decision Process: a decision-making framework where an agent acts, the world changes, and the agent only sees a partial signal (observation) about the true state, not the state itself.
• Admissible action — a generated action (code) that parses and executes without error, regardless of whether it actually solves the task.
• Success Rate (SR) — the percentage of task episodes where the final reward equals 1 (task fully completed).
• Error % — the percentage of an agent’s issued actions that were inadmissible (didn’t run at all), independent of task success.
• Docker container — a lightweight, isolated virtual environment that packages an OS, dependencies, and file system so code can run safely and identically on any machine.
• Gold command/output — the reference (correct) code or answer a dataset provides, used to compute rewards.
• Jaccard Index / Intersection over Union (IoU) — a similarity score between two sets: size of their overlap divided by size of their union; used here to compare an agent’s SQL result rows to the gold rows.
• Kendall’s τ — a statistic measuring how well-ordered one ranked list is relative to another; used here to penalize correctly-selected but wrongly-ordered SQL results.
• ReAct — a prompting strategy that interleaves “Thought” (reasoning text) and “Action” (a tool call/code) steps, letting a model reason about what to do before doing it, then adapt based on the observation.
• Plan & Solve — a prompting strategy that has the model first write an explicit multi-step plan, then execute that plan step by step.
• Try Again — this paper’s own baseline interactive strategy: just keep feeding execution feedback back to the model with no special reasoning scaffold, up to n turns or until success.
• Admissibility / lexical similarity / md5sum — respectively: whether an action ran at all; a text-overlap-based similarity score for comparing two blocks of output; a content hash used here to verify a changed file’s contents (not just its existence) match the gold outcome.