Let It Flow: Agentic Crafting on Rock and Roll — Building the ROME Model within an Open Agentic Learning Ecosystem

TL;DR

Training an LLM to do things over many turns — run commands, read output, fix its mistakes, finish a coding task — is much harder than training it to answer a single prompt. The reward only arrives at the very end (did the tests pass?), trajectories are thousands of tokens long, and the standard RL machinery built for single-turn reasoning becomes unstable at this scale. This paper’s team (Alibaba’s iFlow group) does two things. First, they release a complete ecosystem — ROLL (the RL trainer), ROCK (a secure sandbox that runs the agent’s actions), and iFlow CLI (the agent harness) — so the whole loop from data → training → deployment is one integrated system. Second, they propose IPA (Interaction-Perceptive Agentic Policy Optimization), an RL algorithm whose key move is to stop treating each token as the unit of credit assignment and instead use the interaction chunk (a span of generation that ends in one tool call). The payoff: their 30B model ROME hits 57.4% on SWE-bench Verified and 24.7% on Terminal-Bench 2.0, beating same-size models and rivaling 100B+ ones. They also ship Terminal Bench Pro, a harder, contamination-controlled benchmark, where everyone (including ROME) still struggles — an honest signal that long-horizon agentic tasks are far from solved.

Problem & Motivation

The concrete pain: you cannot reliably RL-train an agent on long, tool-using tasks with the algorithms that work for chat or single-turn math reasoning.

Walk through why. An agentic coding task looks like: read the repo, run a command, see an error, edit a file, run tests, see they fail, edit again, run tests, pass. That is one trajectory — interleaved actions (the model’s tokens) and observations (the environment’s responses) — and it can run thousands of tokens across dozens of turns. The reward is sparse and terminal: you get +1 only if the final unit tests pass, 0 otherwise. Now try to learn from that:

• Credit assignment is broken. RL needs to know which of the model’s decisions deserve credit for the eventual success. Standard methods assign credit at the token level. But the overwhelming majority of tokens (prose, reasoning, formatting) have zero external effect — only the token that triggers a tool call actually changes the world. Spreading credit uniformly over thousands of tokens drowns the signal.
• Discounting collapses. Classic RL down-weights distant actions with a discount factor γ < 1. Over thousands of tokens, γ^3000 ≈ 0 — every early decision vanishes. So practitioners just set γ=1, which throws away all temporal structure and leaves early-state value estimates wildly noisy.
• Off-policy instability. Industrial RL is asynchronous: you generate trajectories with a fast inference engine (SGLang) while training on a different engine (Megatron). The data you train on is stale relative to current weights, and the two engines don’t even produce identical probabilities. This “off-policy” gap, uncorrected, biases the gradient and can collapse the policy.
• Sampling is wasteful. On hard tasks the model almost never succeeds from a cold start, so a batch of rollouts contains almost no positive examples — there’s nothing to reinforce.

Prior work patched these with ad-hoc RL recipes or plain SFT on human demos, but nothing closed the full loop and stayed stable over long horizons. And critically, the open-source community had no end-to-end ecosystem — no integrated data-gen + sandbox + trainer — so reproducing production-grade agent training was nearly impossible.

What’s New (Core Contribution)

Four genuine contributions, separated from the marketing:

1. IPA: chunk-level credit assignment (the real algorithmic novelty).

   • Before: RL for agents assigned advantage/return at the token level (e.g., GRPO/REINFORCE token-wise) or at best per-sentence.
   • Now: The unit of optimization is the interaction chunk — a contiguous span “reason → format the API call → trigger execution” that ends in exactly one tool invocation. Returns, importance-sampling ratios, and masking all operate at chunk granularity. This re-enables temporal discounting (because chunk count K ≪ token count T) and aligns credit with the points where the agent actually affects the environment.

2. A specialized off-policy REINFORCE baseline for industrial RL.

   • Before: PPO-style methods (need a value net + clipping) or naive REINFORCE (unstable off-policy).
   • Now: REINFORCE + geometric-mean Truncated Importance Sampling applied only to negative samples (following TOPR). Positive examples get a clean return-weighted supervised update; negatives get clipped IS so they can’t blow up the policy. This is a deliberate asymmetry — most papers treat positives and negatives identically.

3. Chunk-Level Initialized Resampling (a curriculum trick for sparse rewards).

   • Before: Roll out every trajectory from the initial state; on hard tasks you get ~zero positive signal.
   • Now: Prefill the interaction history with expert-like chunks, then resample only the tail. “Sequential Rollback” and “Parallelized Initialization” find the crucial forks (decision points that determine success) and let the model learn the hard part first, then roll back toward the start — chunk-level curriculum learning. A hybrid IL+RL objective injects a “recovery signal” when no positive trajectory exists at all.

4. The ALE ecosystem + Terminal Bench Pro (systems + evaluation).

   • ROLL (trainer), ROCK (sandbox/execution engine), iFlow CLI (agent harness) released as integrated open infrastructure; ROME (30B MoE on Qwen3) released as the resulting model trained on 1M+ trajectories. Terminal Bench Pro adds 400 hand-authored, contamination-controlled, deterministic terminal tasks across 8 domains — fixing the small-N, leaky, network-flaky problems of existing terminal benchmarks.

How It Works (Technically)

The heart is IPA. Everything else (ROLL/ROCK/iFlow) is the plumbing that lets IPA run at scale. I’ll build IPA up in the same order the paper does: baseline → chunked MDP → chunk-level objective → smarter sampling.

The big picture: the agentic RL loop

flowchart LR
  subgraph Rollout["Rollout (SGLang inference)"]
    A[Agent LLM emits tokens = action] --> B[ROCK sandbox executes tool]
    B --> C[Observation returned]
    C --> A
  end
  C --> D[Episode ends: run unit tests]
  D --> E["Reward R: +1 if all tests pass, else 0"]
  E --> F[Partition trajectory into interaction chunks]
  F --> G["IPA: chunk-level return, IS, masking"]
  G --> H[Megatron training: update weights]
  H -->|sync weights periodically| A

The three systems map onto this loop: iFlow CLI orchestrates the context between LLM and tools (the Rollout box), ROCK is the secure sandbox that actually runs the actions and validates results, and ROLL is the distributed trainer that turns trajectories + rewards into weight updates. They run asynchronously — rollout on some GPUs, training on others — which is what creates the off-policy problem IPA must fix.

Step 1 — The off-policy REINFORCE baseline (demystifying the equations)

Plain REINFORCE has gradient:

∇J = E[ R(τ) · ∇log π(τ) ]

In English: for a whole trajectory τ, scale the gradient of its log-probability by its reward R. If the trajectory succeeded (R>0), push the policy to make those tokens more likely; if it failed, the term is just 0 (binary reward), so nothing happens. It’s a “bandit” view — treat the entire multi-thousand-token trajectory as one action. Simple, no value network, no clipping. That simplicity is exactly why they start here.

The problem: the data was generated by an old/different policy (stale weights + a different inference engine). Training on it as-is gives a biased gradient. The fix is importance sampling (IS) — reweight each sample by how likely the current policy is to have produced it vs. the policy that did:

ρ(τ) = ( ∏ₜ π_new(τₜ | τ<ₜ) / π_old(τₜ | τ<ₜ) )^(1/|τ|)

In English: the ratio of new-policy probability to old-policy probability, multiplied across all tokens, then take the geometric mean (the ^(1/|τ|) exponent). Why geometric mean instead of the raw product? A raw product over thousands of tokens explodes or vanishes the moment one token has a weird ratio. The geometric mean is a per-token average ratio — robust to a single outlier token. They then clip/truncate this ratio (TIS) to cap variance.

The clever asymmetry — their objective splits samples into positive set T⁺ and negative set T⁻:

∇J = Σ_{T⁺} μ_old(τ)·R(τ)·∇log π_new(τ) + Σ_{T⁻} μ_old(τ)·[ρ(τ)]·R(τ)·∇log π_new(τ)

In English: positive trajectories get a straight return-weighted supervised-learning push (no IS dampening — you trust good data and learn fast from it). Negative trajectories get the clipped IS treatment, because a flood of negative samples is what causes “policy collapse” — probability mass getting squeezed onto useless tokens. So: lean into wins, be careful with losses.

They also mask tokens where the inference engine and training engine disagree too much (importance weight above a threshold H) — those tokens are untrustworthy, so zero out their gradient.

Step 2 — The Chunked MDP (the conceptual core)

Instead of modeling the task as a token-by-token Markov Decision Process, they partition each trajectory τ[1:T] into chunks {c₁,…,c_K} where K ≪ T. Each chunk = “one functional unit ending in a tool call” (reason → build the call → execute). The MDP tuple becomes (S, C, P, R, γ) where:

• S = states, each encoding the full interaction history up to the start of a chunk.
• C = chunk-actions (a variable-length token span ending in a tool call or task completion).
• P = transitions, governed by the LLM’s generation and the external tool’s stochastic response.
• R = sparse terminal reward (tests pass).
• γ = discount factor, now applied per chunk.

Why this matters: token-level granularity mismatches reality (most tokens have no external effect); sentence-level is still wrong (one tool call often spans several sentences, only the last triggers anything). Chunk = the natural causal unit.

Schematic: the same trajectory viewed at token, sentence, and chunk granularity. Watch how chunk boundaries land exactly on tool-call points — the only places the environment actually changes. Click to step through granularities.

Step 3 — The chunk-level objective (where the magic lands)

Three refinements, all moved from token-scale to chunk-scale:

(a) Discounted Chunk-Level Return. For chunk c_k:

G_k = γ^Δ(j,k) · R_final

In English: Δ(j,k) is how many chunks separate this chunk from the chunk that finished the task. Because there are only K≈dozens of chunks (not thousands of tokens), γ^Δ doesn’t vanish. Chunks right before success get γ^Δ ≈ 1 (strong gradient); early useless chunks (e.g., a botched tool call) get exponentially suppressed. Every token inside a chunk shares that chunk’s scalar G_k. This is the line that fixes the “discounting collapses over thousands of tokens” problem — shorten the effective horizon and discounting becomes meaningful again.

(b) Chunk-Level Importance Sampling. Same geometric-mean IS as before, but the product runs over tokens within a chunk and is averaged by chunk length |c|. Coarser horizon → fewer extreme ratios → more stable.

(c) Chunk-level masking. The mismatch mask is relaxed to the chunk horizon rather than per-token, avoiding over-aggressive gradient suppression.

Putting it together, the full IPA gradient (Eq. 7) is the off-policy split objective from Step 1, but with G_c (chunk return) replacing R(τ), chunk-IS ρ_c replacing token-IS, and the gradient summed over chunk log-probs:

∇J = Σ_{T⁺} μ_old(c)·G_c·Σ_k m_c ∇log π(c_k) + Σ_{T⁻} μ_old(c)·[ρ_c(c)]·G_c·Σ_k m_c ∇log π(c_k)

Empirically (their Fig 10) this gives smoother gradient norms and higher train- and test-time success vs. the token-level baseline.

Step 4 — Chunk-Level Initialized Resampling (fixing sparse rewards)

On hard tasks, success from a cold start is near-zero, so batches have no positives to reinforce. Their insight: prefill the history with correct expert chunks and resample only the rest. Mechanics:

• Identify crucial forks — a chunk c_f is crucial if the expected success rate of resampling after it is far higher than before it (i.e., that decision is decisive and the policy hasn’t mastered it).
• Sequential Rollback: start resampling from the last chunk of an expert trajectory, move backward. Tail states need few turns to finish → positives are easy → learn the end first, then roll back. This is chunk-level curriculum learning.
• Parallelized Initialization: Sequential Rollback is slow if the decisive fork is early. So instead, pick anchor chunks at several positions and launch parallel rollouts from each — trades sample density per fork for speed.
• Hybrid IL+RL fallback: if no positive trajectory exists at a fork, pure RL gives zero gradient. So add an imitation-learning term on the expert’s chunks (a “recovery signal,” balanced by λ_IL and λ_RL) to keep the policy anchored in good regions and prevent collapse.

The algorithm, simplified

# IPA core: one policy-gradient step over a batch of agentic trajectories.
# Hidden: model calls, sandbox execution, tokenization. Exposed: the chunk logic.

GAMMA, IS_CLIP, H = 0.95, 10.0, 5.0   # discount, IS clip, mismatch-mask threshold

def ipa_step(trajectories):            # each traj has tokens, tool-call boundaries, final reward
    grad = 0
    for traj in trajectories:
        chunks = split_on_tool_calls(traj)        # span = reason->build call->execute  (K << T)
        K = len(chunks)
        R_final = traj.reward                      # +1 iff all unit tests passed, else 0
        for k, c in enumerate(chunks):
            G = GAMMA ** (K - 1 - k) * R_final     # chunk return: late chunks ~undiscounted
            # geometric-mean importance ratio over tokens IN this chunk (robust to outliers)
            rho = geomean(pi_new(t) / pi_old(t) for t in c.tokens)
            mask = [1 if mismatch(t) < H else 0 for t in c.tokens]   # drop untrustworthy tokens
            logp = sum(m * log_pi_new(t) for m, t in zip(mask, c.tokens))
            if R_final > 0:                        # positive: trust it, fast SL-style update
                grad += mu_old(c) * G * logp
            else:                                  # negative: clip IS so it can't collapse policy
                grad += mu_old(c) * min(rho, IS_CLIP) * G * logp
    return grad / len(trajectories)

def hard_task_curriculum(task, expert_traj):
    # prefill expert chunks, resample the tail -> positives become reachable
    fork = find_crucial_fork(task, expert_traj)    # where success-rate drops sharply
    rollouts = resample_from(state_before=fork)     # Sequential Rollback / Parallel Init
    if not any(r.success for r in rollouts):        # no positive signal at all?
        return imitation_loss(expert_traj, up_to=fork)   # IL fallback "recovery signal"
    return rollouts

Built on Prior Work

Results & Evidence

Headline numbers (ROME, 30B MoE / 3B active):

• SWE-bench Verified: 57.4% — beats GLM-4.5-Air (56.2%, 106B), GPT-OSS-120B (43.9%), Gemini-2.5-Flash (28.7%); near GPT-5-Mini (59.3%).
• Terminal-Bench 2.0: 24.7% — best among the same-size and several larger models in their table; overall terminal avg 37.6% edges GPT-5-Mini’s 37.99%-class results.
• Competitive on tool-use (TAU2, BFCL-V3, MTU) and general agentic (GAIA, BrowseComp-ZH, ShopAgent) suites.
• Ablations (their Figs 10, 12, 13) show chunk-level optimization → smoother gradient norms + higher success than token baseline; chunk-initialized resampling lets the model solve tasks it otherwise never solves.

What the evidence establishes: chunk-level credit assignment is a real, measurable stabilizer for long-horizon agentic RL, and a well-engineered 30B model can punch at 100B+ weight class on these specific benchmarks. Production deployment in iFlow CLI is a genuine signal it survives contact with real use.

What it does NOT establish / caveats:

• Everyone scores low on Terminal Bench Pro — including ROME. The authors are honest: error compounding, weak recovery, brittle long-term planning persist regardless of scale. So “rivals 100B models” means “on benchmarks that may be near saturation/leaky,” not “solved agentic crafting.”
• No clean isolation of IPA vs. ecosystem vs. data. ROME’s win bundles 1M+ curated trajectories, a strong base model, the systems, and IPA. The ablations show IPA helps on mini-sets, but the end-to-end “57.4%” can’t be attributed to IPA alone.
• Self-reported, partly leaderboard-sourced numbers (the * entries), Avg@3 — reasonable but not third-party audited.
• Massive compute/data (1M+ trajectories, async multi-engine cluster) — not reproducible by a small team without the released infra actually being usable end-to-end.

How You’d Use It

For an AI services shop, this paper is more useful as an architecture blueprint and a set of training tricks than as a model you’d deploy (though ROME is open and worth benchmarking).

• Adopt the chunk abstraction in your own agent evals and RL/fine-tuning. Even if you’re not doing RL, “segment the trajectory by tool call” is the right unit for analyzing where agents fail, for reward shaping, and for building datasets. It maps cleanly onto a ReAct-style loop you’ve already built.
• Steal the off-policy stability recipe. If you ever do RL/DPO-style tuning on agent trajectories asynchronously, the “trust positives, clip negatives, geometric-mean IS” pattern is a concrete, low-risk stabilizer.
• Use the resampling/curriculum idea for synthetic data. Prefilling expert chunks and resampling tails is a cheap way to manufacture positive trajectories on hard tasks — directly applicable to SFT dataset generation, no RL required.
• Sandbox-as-a-service. ROCK’s “secure sandbox that executes agent actions + validates with unit tests” is exactly the component clients underestimate. Standing up a hardened execution environment with permission control is a sellable capability and a moat.
• Benchmark clients’ agents on Terminal Bench Pro — a contamination-controlled, deterministic eval is a credible deliverable for “is this agent actually good?” engagements.

Build Your Own (Minimal Recipe)

Smallest version that captures ~80% of the value — chunk-level credit assignment for agent fine-tuning, skipping the industrial async cluster:

1. Agent harness + sandbox. A ReAct loop (reason → tool call → observation) running tools inside a Docker sandbox with a test-based reward. Reach for: any agent framework you already use + Docker; reward = pytest exit code.
2. Trajectory logger that records chunk boundaries. Every time the agent emits a tool call, close a chunk. Store (state, chunk_tokens, tool_result). This is the one piece most stacks don’t have — add it first.
3. Collect trajectories with terminal rewards. Run your agent on a task set; tag each trajectory pass/fail.
4. Chunk-level objective. Compute G_k = γ^(K-1-k)·R_final per chunk; weight each chunk’s token log-probs by G_k. Start with plain policy gradient (the ipa_step above) before adding IS.
5. (Optional) off-policy correction only if you go async: add geometric-mean clipped IS on negatives.
6. (Optional) resampling curriculum for hard tasks: prefill expert chunks, resample tails.

The two genuinely hard parts: (a) the secure, reproducible sandbox — isolation, determinism, no network flakiness — this is where most effort goes; (b) stable RL infra — async rollout/train, weight sync, KV-cache management. If you don’t need RL, do steps 1–4 as supervised chunk-weighted fine-tuning and you avoid (b) entirely. Libraries to reach for: their released ROLL/ROCK, or TRL/verl for RL, vLLM/SGLang for inference, a Qwen3-MoE or similar open base.

How to Improve It

1. Learned chunk boundaries. Chunks are currently defined by tool-call positions. Some decisive reasoning happens within a chunk before the call. Train a lightweight segmenter (or use attention/entropy spikes) to place boundaries at true decision points, not just tool calls.
2. Per-chunk learned advantage instead of a shared scalar G_k. Every token in a chunk shares one return; a small per-chunk critic (a cheap value head) could give intra-chunk credit and reduce variance further — test whether it beats the discount-only scheme.
3. Process rewards / verifier on intermediate chunks. R is purely terminal. Add a learned or rule-based verifier that scores chunk-level subgoal completion (compiles? test count improved?) to densify the reward without human labels.
4. Auto-discover crucial forks cheaply. Sequential Rollback is O(K) rollouts per task. A learned “fork predictor” trained on past resampling drops could jump straight to decisive chunks, cutting curriculum cost dramatically.
5. Attack the Terminal Bench Pro failure modes directly. The paper names them: error compounding, weak recovery, brittle planning. Combine IPA with an explicit reflection/backtracking memory (Reflexion-style verbal feedback persisted across chunks) and measure whether recovery improves — this is the open frontier the authors themselves flag.

Glossary

• Agentic crafting — multi-turn, tool-using LLM workflows (run command → observe → refine), as opposed to one-shot answer generation.
• Trajectory (τ) — the full sequence of interleaved agent actions (tokens) and environment observations for one task attempt.
• Interaction chunk — a contiguous span of generation that ends in exactly one tool call; IPA’s unit of credit assignment. K chunks ≪ T tokens.
• Credit assignment — deciding which of the agent’s decisions deserve credit/blame for the final outcome; the central RL problem here.
• Sparse/terminal reward — you only get a score at the very end (tests pass = 1, else 0); nothing in between.
• REINFORCE — the simplest policy-gradient RL: scale log-prob gradients by trajectory reward; no value network.
• PPO / GRPO — popular RL algorithms for LLMs; PPO uses a clipped surrogate + value net, GRPO uses group-relative advantages. The paper deliberately uses neither’s full machinery.
• Off-policy — training on data generated by an older/different policy than the one you’re updating; efficient but biased without correction.
• Importance sampling (IS) — reweighting off-policy samples by π_new/π_old to correct the bias; high variance unless clipped.
• Geometric-mean IS — taking the |τ|-th root of the product of per-token ratios; robust to a single outlier token vs. a raw product.
• TIS (Truncated IS) / clipping — capping the IS ratio to bound gradient variance.
• Discount factor (γ) — down-weights distant decisions; useless over thousands of tokens, meaningful over dozens of chunks.
• Chunked MDP — the task modeled as a Markov Decision Process whose actions are chunks, not tokens.
• Crucial fork — a decision point (chunk) where the agent’s choice disproportionately determines success.
• Sequential Rollback / Parallelized Initialization — strategies to start rollouts from expert-prefilled tail states so positive examples become reachable on hard tasks.
• Imitation learning (IL) fallback — supervised copying of expert chunks when RL produces zero positive signal, preventing policy collapse.
• ROLL / ROCK / iFlow CLI — the trainer, the sandbox execution engine, and the agent harness making up the ALE ecosystem.
• ROME — the released open agent model (30B-parameter MoE, ~3B active, built on Qwen3-MoE) trained via this pipeline.
• MoE (Mixture of Experts) — architecture where only a subset of parameters (here ~3B of 30B) activate per token, giving large-model capacity at small-model inference cost.
• SGLang / Megatron-LM — the inference engine (rollout) and training engine, respectively; their mismatch is a source of off-policy instability.
• Terminal Bench Pro — the paper’s new 400-task, deterministic, contamination-controlled benchmark for terminal agents across 8 domains.