Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

TL;DR

Multimodal LLMs (models that take an image + a text question) are good at perceiving but bad at long multi-step reasoning, mostly because nobody has a cheap way to produce high-quality long-chain reasoning data for images, and because naively fine-tuning one model on long reasoning chains actually makes it worse. Insight-V fixes both. First, a fully automated pipeline generates structured, multi-step reasoning traces (each step has a title, a detailed thought, and a continue/summary action) and then scores them with other models, so you get a scalable dataset with zero human annotation. Second — and this is the real insight — they split the work into two agents: a reasoning agent that produces a long detailed chain of thought, and a summary agent trained to critically read that chain (which may contain mistakes) and decide the final answer. They then sharpen the reasoning agent with iterative DPO (a reinforcement-learning-style preference method). On seven hard visual-reasoning benchmarks this lifts LLaVA-NeXT by an average of 7.0 points and a stronger base model by 2.9, without hurting basic perception tasks.

Problem & Motivation

The pain in one sentence: you can’t make a vision model reason in long chains because (1) there’s no cheap source of long visual-reasoning training data, and (2) when you do force long chains through a single model, accuracy drops because errors compound.

Two concrete obstacles:

• Data is expensive and absent. Text-only reasoning (math, code) has tons of structured chain-of-thought data and proven training recipes. Visual reasoning data — “look at this chart/table/scene and reason step by step to an answer” — is much more expensive to collect and demands heavy human annotation. There’s no established auto-generation pipeline, so the field is starved.
• One model doing everything is fragile. Prior work showed that just bolting chain-of-thought onto an MLLM gives only modest gains, and other work found current training barely improves CoT at all. The deeper issue: when a single model has to both generate a long reasoning chain and commit to a final answer, a wrong step early in the chain drags the final answer down with it. The model can’t “step back” and judge its own flawed reasoning — it’s too entangled in producing it. The longer the chain, the more it falters.

So the question Insight-V attacks is: how do you (a) manufacture good long-chain visual reasoning data at scale, and (b) train a model to actually benefit from reasoning long instead of being hurt by it?

What’s New (Core Contribution)

Three contributions, each a “before → now”:

1. Scalable, human-free reasoning-data pipeline. Before: long-chain visual reasoning datasets needed human annotators to write and validate step-by-step solutions. Now: a reasoning generator emits structured JSON reasoning (step summary + detailed thought + next action) progressively, sampling N diverse chains per question, then a two-stage multi-granularity assessment (LLM answer-correctness filter → multimodal step-quality scorer, 1–100) keeps only good chains. No humans in the loop.

2. A reasoning/summarization multi-agent decomposition. Before: one MLLM both reasons and answers, so flawed reasoning poisons the answer. Now: a reasoning agent specializes in producing the long detailed chain, and a separate summary agent is explicitly trained to be robust to errors in that chain — it critically reads the reasoning, ignores bad parts, and decides the answer. Crucially, both agents are fine-tuned from the same base model (it’s a role split, not two different architectures).

3. Iterative DPO on the reasoning agent. Before: offline DPO preference data goes stale as the model changes. Now: run DPO in rounds — the current model generates fresh preference pairs for the next round — approximating online RL and progressively refining reasoning quality. They show their own model-generated preference data beats an off-the-shelf alignment dataset (RLAIF-V).

The genuinely new idea is #2: the field had CoT and had DPO; the summary-agent-as-critic split, plus deliberately training it on flawed chains so it learns to judge rather than copy, is what makes long reasoning help instead of hurt.

How It Works (Technically)

There are three machines here: the data pipeline, the two-agent system at inference, and the two-stage training (SFT then iterative DPO). Let’s trace one question all the way through.

Take the running example from Figure 1: an image of a baseball game and the question “How many people are visible in this picture?”

1) Generate the reasoning data (offline, to build the training set). For each training question, a reasoning generator model produces a structured response step by step. Each step is a JSON object with three fields: a short summary of the step, a detailed reasoning body, and an action that is either continue (do another step) or summary (you’re done, write the final answer). Formally the paper writes:

• R_t = M(I, Q, [R_1 ... R_{t-1}], A) — “the response at step t is produced by model M given the image I, the question Q, all prior steps, and the action A decided last step.” In plain terms: each step is conditioned on everything that came before, so the chain grows coherently.
• R_ans = M(I, Q, [R_1 ... R_n]) — once the action is summary, the model produces the final answer from the full chain of n steps.

They repeat this whole process N times per question with sampling turned up, to get N diverse chains (different lengths, different details). This diversity is the point — you want to discover which reasoning path actually works for each question.

2) Assess and filter (multi-granularity). N raw chains per question is noisy. Two filters:

• Answer filtering (coarse): a strong LLM (Qwen2) compares each chain’s final answer to the ground truth and throws out chains that got it wrong. Right answer ≈ probably sane reasoning — a cheap proxy.
• Reasoning-path scoring (fine): a strong multimodal model (Qwen2-VL) gets the image + question + surviving chain + ground truth and scores each chain 1–100 on step-by-step correctness and level of detail. To keep scores comparable, all chains for one question are scored in a single pass.

The output is a structured dataset where each question has reasoning chains with quality scores. The reasoning agent is trained on the highest-scoring chain per question. The summary agent is trained on a deliberately mixed bag — good chains and low/mid-scoring flawed chains — so it learns to critically judge, not blindly copy.

3) Inference with two agents. At test time:

• The reasoning agent reads (image, question) and emits the long structured chain — step 1 “identify key info,” step 2 “count main figures,” step 3 “revisit background,” step 4 “estimate total,” then action summary.
• The summary agent reads (image, question, the chain) and produces the final answer. Because it was trained on flawed chains too, if the reasoning over-counted or contradicted itself, the summary agent can down-weight or discard that and still answer correctly. This is the “robust to inaccuracies” property — and it’s why the decomposition beats a single model.

4) Sharpen the reasoning agent with iterative DPO. SFT alone makes a decent reasoning agent; DPO makes it better-aligned and more stable. Quick demystification of the math, because the reader profile asks for it:

• DPO works on preference pairs (x, y_w, y_l): a prompt x, a preferred (winning) response y_w, and a dispreferred (losing) one y_l. The notation y_w ≻ y_l | x just means “for prompt x, w is preferred over l.”
• The Bradley-Terry model turns a pair of scalar “rewards” into a probability that one wins: p(y_1 ≻ y_2 | x) = σ(r(x,y_1) − r(x,y_2)), where σ is the logistic/sigmoid function. Read it as: the bigger the reward gap, the more confidently the better one wins; equal rewards → 50/50.
• You fit the reward model by minimizing L = −E[log σ(r(x,y_w) − r(x,y_l))] — “push the winner’s reward above the loser’s.” DPO’s trick (from Rafailov et al.) is that you never train a separate reward model; the language model is the reward model, so you can optimize this preference objective directly on the policy. That’s why it’s simpler than full RLHF/PPO — no separate reward net, no rollout loop.
• Iterative DPO: ordinary DPO uses one fixed offline set of preference pairs. As the model drifts during training, those pairs no longer reflect what the model would actually generate, so the signal weakens. Insight-V trains a sequence M_1 → M_2 → … where each M_{t+1} learns from preference data D_t generated by M_t. Each round’s data is “on-distribution” for the current model, which approximates online RL. They do 3 rounds on ~15K pairs at a tiny learning rate (5e-7).

Architecture & data flow

flowchart TD
  subgraph DATA[Offline: Data Generation Pipeline]
    Q[Image + Question] --> GEN[Reasoning Generator<br/>N diverse JSON chains]
    GEN --> AF{Answer Filter<br/>LLM vs ground truth}
    AF -->|wrong| X[Discard]
    AF -->|correct| SC[Reasoning-Path Scorer<br/>Qwen2-VL, 1-100]
    SC --> RD[Reason Dataset<br/>best chain per Q]
    SC --> SD[Summary Dataset<br/>good + flawed chains]
  end

  subgraph TRAIN[Training: from ONE base model]
    BASE[Base MLLM] --> RSFT[SFT --> Reasoning Agent]
    BASE --> SSFT[SFT --> Summary Agent]
    RD --> RSFT
    SD --> SSFT
    RSFT --> DPO[Iterative DPO x3<br/>self-generated prefs]
  end

  subgraph INFER[Inference: Two Agents Collaborate]
    IN[Image + Question] --> RA[Reasoning Agent<br/>long structured chain]
    RA --> SA[Summary Agent<br/>judges chain, answers]
    SA --> OUT[Final Answer]
  end

  DPO --> RA
  SSFT --> SA

Schematic of the two-agent inference loop. Click "step" to advance the reasoning agent one step at a time; the summary agent then reads the whole (possibly flawed) chain and commits to an answer. Watch how a wrong intermediate step doesn't necessarily corrupt the final answer.

The algorithm, simplified

The heart is the two-agent inference plus the data-curation logic. Here’s the whole thing in toy form:

# llm(model, image, question, context) -> str : one MLLM generation call

def generate_reasoning_data(image, question, ground_truth, N=8):
    chains = []
    for _ in range(N):                       # sample N diverse chains (high temp)
        steps, action = [], "continue"
        while action == "continue":
            step = llm(generator, image, question, steps)  # {summary, reasoning, action}
            steps.append(step)
            action = step["action"]          # model decides when to stop
        answer = llm(generator, image, question, steps)    # final answer
        chains.append({"steps": steps, "answer": answer})

    # multi-granularity assessment
    correct = [c for c in chains if judge_correct(c["answer"], ground_truth)]  # coarse filter
    for c in correct:
        c["score"] = score_path(image, question, c, ground_truth)  # fine: 1..100

    reason_data  = max(correct, key=lambda c: c["score"])          # best chain -> reasoning agent
    summary_data = sample_across_score_bands(correct)              # good + flawed -> summary agent
    return reason_data, summary_data


def insight_v_infer(image, question):
    chain = reasoning_agent(image, question)        # long step-by-step reasoning (may have errors)
    # summary agent was trained on FLAWED chains too, so it judges instead of copying:
    answer = summary_agent(image, question, chain)  # robust to inaccuracies in `chain`
    return answer

The two non-obvious moves: (1) you let the generator decide when to stop via the action field rather than fixing a step count, and (2) you intentionally feed the summary agent broken reasoning so it learns the skill of critical evaluation, not transcription.

Built on Prior Work

Results & Evidence

Headline numbers (7 visual-reasoning benchmarks: MMMU, MMMU-Pro, MMBench, MME, ChartQA, MMStar, MathVista):

• Applied to LLaVA-NeXT-LLaMA3 (8B): average 40.2 → 47.2, +7.0. Multi-agent alone gives +4.3 (40.2→44.5); iterative DPO adds the rest.
• Applied to their stronger 7B base model: 48.7 → 51.6, +2.9.
• Biggest single jump: MME +9.1 (perception+cognition); ChartQA +5.8; MathVista +3.0.
• Perception isn’t sacrificed: on TextVQA/DocVQA/OCRBench/AI2D, scores improve too (e.g., OCRBench 553→663 on LLaVA), suggesting the reasoning agent helps the model attend to relevant regions.

Ablations that actually matter (this is where the claims earn their keep):

• Multi-agent vs. alternatives (Table 3): multi-agent avg 62.1 beats “vanilla direct SFT” CoT (60.6), “multi-turn supervised” (61.0), and “summary agent only / no reasoning” (59.8). So the decomposition is doing real work, not just more data.
• Data scaling (Fig 4): the reasoning agent needs scale — with too little data it underperforms the baseline (a bad chain is worse than no chain), and it improves monotonically from 50K→200K. Honest and important: this method has a minimum data threshold to pay off.
• DPO strategy (Table 4): their curated DPO data (+0.6 avg) beats RLAIF-V (+0.2) at equal size; iterative adds another +0.6.

Caveats / what the evidence does NOT establish:

• All gains are on 7–8B open models; no evidence it scales to frontier-size models or that the summary-agent trick still helps when the base reasoner is already very strong (note the +2.9 on the stronger base vs +7.0 on the weaker one — diminishing returns).
• The assessment pipeline trusts other models (Qwen2 / Qwen2-VL) as graders. “Right final answer” is a loose proxy for “good reasoning” — a chain can be right for the wrong reasons and still get kept.
• Inference now costs two forward passes (reason, then summarize) plus a long chain — meaningfully more tokens/latency than a single answer. The paper doesn’t foreground the compute cost.
• Benchmarks are standard but the gains, while consistent, are single-digit on most tasks; this is a solid engineering win, not a step-change.

How You’d Use It

For an AI-services shop, this is a recipe you can resell as a capability, not a model you have to license. Concretely:

• “Reasoning upgrade” as a service. A client has a fine-tuned or open MLLM doing document/chart/diagram Q&A and it’s flaky on multi-step questions. You apply the Insight-V recipe: auto-generate reasoning data from their domain images (invoices, schematics, medical charts, dashboards), fine-tune a reasoning + summary agent pair from their base model. Deliverable: measurably better accuracy on the hard subset, perception unharmed.
• The summary-agent-as-verifier pattern maps straight onto MAS work you’ve already built. You’ve run a multi-agent system; this is the same coordination idea with a sharp twist: train one agent to be robust to a teammate’s mistakes rather than trusting the upstream output. That “critic that’s been deliberately trained on bad inputs” is a reusable design for any pipeline where one stage is noisy (OCR → parse, retrieval → answer, plan → execute).
• Synthetic data engine. Even if you skip the two-agent inference, the generation+assessment pipeline is a standalone product: a way to mint scored chain-of-thought datasets for any vertical with no labelers. That’s directly sellable as “we build your reasoning dataset.”
• Where it slots in: it’s a training-time enhancement, so it’s behind your inference API. Budget for the extra inference cost (two passes) and decide per-client whether the accuracy is worth the latency.

Realistic effort: if the client already has a trained base MLLM and GPUs, the pipeline + two SFT runs + a few DPO rounds is a weeks-not-months engagement. The hard part is the grader quality (see Build).

Build Your Own (Minimal Recipe)

The 80/20 version — you can skip iterative DPO entirely and still capture most of the gain (the ablation shows multi-agent alone is most of it).

Components, in build order:

1. A base MLLM you can fine-tune — start with an open LLaVA-NeXT or Qwen2-VL checkpoint. Don’t train one from scratch.
2. The generator loop. Prompt the base (or a stronger MLLM) to emit JSON {summary, reasoning, action} per step with action ∈ {continue, summary}. Sample N=4–8 chains per question at high temperature. This is just structured prompting + a while-loop.
3. The two-stage grader. (a) An LLM that compares final answer to ground truth (you need labeled Q/A, even if reasoning is unlabeled). (b) A strong multimodal model that scores the chain 1–100. Both are off-the-shelf API/model calls.
4. Two SFT datasets. Reasoning agent ← top-scoring chain per question. Summary agent ← (image, question, chain) → answer using a mix of good and deliberately flawed chains, plus some plain VQA data to preserve perception.
5. Two LoRA/full fine-tunes from the same base, then wire the two-call inference: answer = summary(image, q, reason(image, q)).

The 1–2 genuinely hard parts:

• Grader reliability. Garbage scores → garbage reasoning agent. The “right answer = good chain” proxy is weak; budget effort on a better scoring prompt or a small human-validated calibration set.
• Curating flawed summary data correctly. Too-easy errors and the summary agent learns nothing; too-broken and it learns to ignore reasoning entirely. The paper samples flawed chains across score bands — replicate that stratification.

Libraries/models to reach for: Hugging Face transformers + a VLM checkpoint, trl for SFT and DPO (DPOTrainer does exactly the BT-loss math above), peft for LoRA, and any strong VLM (Qwen2-VL, GPT-4o) as the grader.

How to Improve It

Limitations as leverage — five testable directions:

1. Replace the answer-correctness proxy with a process reward model. The coarse filter keeps chains that are “right for the wrong reasons.” Train a step-level verifier (PRM-style) so scoring rewards valid steps, not just correct endpoints. Testable: does PRM-curated data beat answer-filtered data at equal size?
2. Collapse two passes into one with a single reason+verify head. The two-call inference doubles latency. Try a single model with a special “verify/summarize” turn or a learned gate that decides when to trust the chain — measure accuracy retained vs. latency saved.
3. Swap iterative DPO for GRPO/online RL with a verifier reward. DPO needs preference pairs; o1/R1-style training uses a verifier reward directly over sampled chains. Given they already have a scorer, GRPO (sample group, advantage = score − group mean, reinforce) is a natural upgrade and avoids stale-pair issues entirely.
4. Make the summary agent emit a confidence + abstain. Right now it always answers. Train it to say “reasoning insufficient” and trigger another reasoning pass (test-time scaling) — directly attacks the cases where the single chain is wrong.
5. Adaptive chain length / early exit. The generator decides continue vs summary, but there’s no budget control. Add a reward term penalizing length on easy questions (like o1’s compute-proportional-to-difficulty), so you don’t burn tokens reasoning about trivial perception queries.

Glossary

• MLLM (Multimodal LLM) — a language model that also takes images (and sometimes audio/video) as input, e.g., LLaVA, Qwen2-VL.
• Chain-of-Thought (CoT) — prompting/training a model to write out intermediate reasoning steps before the final answer.
• Long-chain reasoning — extended, multi-step reasoning (think o1-style) vs. a one-shot answer.
• Multi-agent system — multiple specialized model instances that collaborate; here, a reasoning agent and a summary agent split from one base model.
• Summary agent — the critic/decider: reads the reasoning chain (errors and all) and outputs the final answer, trained to be robust to flawed reasoning.
• SFT (Supervised Fine-Tuning) — standard training on (input → desired output) pairs; how each agent first learns its role.
• DPO (Direct Preference Optimization) — aligns a model directly on preferred-vs-dispreferred response pairs without training a separate reward model or running PPO.
• Iterative DPO — running DPO in rounds, regenerating fresh preference pairs from the current model each round to approximate online RL.
• Bradley-Terry model — a formula converting two scalar rewards into the probability one beats the other; the statistical backbone of preference learning.
• Reward model — a model that scores how good a response is; in DPO the policy itself implicitly plays this role.
• RLHF / RLAIF — Reinforcement Learning from Human (or AI) Feedback; aligning models to feedback signals. RLAIF-V is an open AI-feedback alignment dataset used here as a baseline.
• Multi-granularity assessment — the two-level grading: coarse answer-correctness filter + fine 1–100 reasoning-path score.
• Process reward model (PRM) — a verifier that scores individual reasoning steps rather than only the final answer (a suggested improvement, not used in the paper).
• GRPO — Group Relative Policy Optimization; an RL method that scores a group of sampled answers and reinforces the above-average ones (suggested as an alternative to DPO).