Latent Collaboration in Multi-Agent Systems

TL;DR

Multi-agent systems (MAS) usually have each LLM write out its reasoning as text, then hand that text to the next agent. That text bottleneck is wasteful: generating tokens is slow, every token throws away most of the information packed in the model’s hidden state, and re-reading another agent’s text means re-encoding it from scratch. LatentMAS removes the text layer entirely. Each agent “thinks” by auto-regressively feeding its own last-layer hidden state back in as the next input (no decoding), and agents communicate by directly transplanting each other’s KV-caches into each other’s attention layers. It is training-free — works on off-the-shelf Qwen3 models — and across 9 benchmarks it beats text-based MAS by +2.8% to +4.6% accuracy, uses 70-84% fewer tokens, and runs ~4x faster end-to-end. The headline insight: a single latent step carries roughly d_hidden / log|V| times more information than a single text token (hundreds of times more for a 4B-14B model), so a handful of latent steps replaces thousands of decoded tokens.

Problem & Motivation

The concrete pain: text is a terrible inter-agent protocol, but everyone uses it because it’s all we have.

When agent A (a “planner”) finishes and passes its plan to agent B (a “solver”), three things go wrong:

1. Decoding is slow and lossy. A transformer’s last-layer hidden state h_t is a dense ~1024-to-5120-dimensional real vector. To emit a token, the model collapses that whole vector down to a single choice from the vocabulary via softmax(h_t · W_out). That projection throws away almost everything — you keep log|V| bits (~17 bits for a 150k vocab) and discard the rest of a high-dimensional continuous representation. Then the next agent has to re-read that lossy text and re-encode it back into hidden states, paying full quadratic attention cost again.

2. Token budgets explode. On a hard reasoning task like AIME, a text-based MAS chain can burn 20,000+ output tokens writing out chain-of-thought for every agent. That’s directly your inference bill and your latency.

3. Prior latent work was incomplete. People had explored latent reasoning inside one model (e.g., Coconut, which loops hidden states back without decoding) and, separately, latent communication between two models (sharing KV-caches of the input prompt). But nobody had unified both into a full multi-agent collaboration framework where agents both think and talk in latent space.

So the question the paper poses verbatim: “Can multi-agent systems achieve pure latent collaboration?”

What’s New (Core Contribution)

Four genuine contributions, each a “before / now”:

• Pure latent collaboration end-to-end. Before: latent reasoning was a single-model trick; cross-model latent sharing only passed the input prompt’s cache. Now: both reasoning AND communication happen in latent space, across an arbitrary number of agents, with only the final agent decoding to text.
• Latent working-memory transfer via full layer-wise KV-caches. Before: cache-sharing methods (e.g., KVComm-style) exchanged only the prefilled input context between models. Now: LatentMAS extracts the KV-cache from all L layers after an agent has done its latent thinking — so the cache carries both the original context AND the newly generated latent thoughts — and prepends it into the next agent’s attention. The paper proves this is lossless: agent B’s output is identical to what it would produce if A had handed over its actual text output (Theorem 3.3).
• A training-free input-output alignment fix (W_a). Before: naively feeding a last-layer hidden state back as an input embedding causes out-of-distribution activations (output space ≠ input-embedding space). Now: a tiny d_h × d_h projection matrix W_a ≈ W_out⁻¹ W_in, computed once via ridge regression, maps outputs back into the valid input-embedding distribution. Cheap, no fine-tuning, +2.3-5.3% accuracy.
• Theory quantifying the win. A formal result (Theorem 3.1) that one latent step is worth Ω(d_h / log|V|) text tokens of expressiveness — 235x to 471x for Qwen3-4B/8B/14B — plus a complexity analysis showing latent collaboration is strictly cheaper than text MAS at equal expressiveness (Theorem 3.4).

How It Works (Technically)

The whole system has two mechanisms — latent thinking (inside an agent) and latent communication (between agents) — plus one stabilizer (W_a).

1. Latent thoughts generation (inside one agent)

Normal autoregressive generation: feed tokens in, get hidden states out, project the last one to vocab logits, sample a token, embed it, repeat.

Latent generation skips the sample-and-embed step. After computing the last-layer hidden state h_t, instead of token = sample(softmax(h_t · W_out)) and then e_{t+1} = embed(token), you just set e_{t+1} = h_t directly (after the W_a fix below) and run the forward pass again. You do this m times. The result is a sequence of latent thoughts H = [h_{t+1}, ..., h_{t+m}] that never touched the vocabulary. Think of it as the model “muttering to itself in its own internal vector language” for m steps before anyone reads anything out loud.

Why this is more expressive — the key equation, demystified:

Theorem 3.1: To losslessly reproduce m latent steps as text, you need at least Ω(d_h · m / log|V|) tokens.

Plain English: each hidden state is a point in d_h-dimensional continuous space. Under the Linear Representation Hypothesis (concepts are roughly linear directions in that space), the information content of one such vector is on the order of d_h units, whereas one token gives you only log|V| bits (the number of bits to name one item from the vocabulary). Divide and you get the ratio. Operationally: replace ~470 decoded tokens with one latent step on a 14B model, in principle. In practice they use m ≈ 40-80 latent steps and match or beat MAS chains that emit 20k+ tokens.

2. Input-output alignment (W_a) — the stabilizer

There’s a subtle bug in step 1. A model’s input embeddings (from W_in) and its output hidden states (from the residual stream, read by W_out) live in statistically different regions. Feed a raw h_t in as if it were an input embedding and you get out-of-distribution activations that degrade over m steps (representation drift).

Fix: find a matrix W_a such that e = h · W_a lands h back in the input-embedding distribution. The ideal would be W_a = W_out⁻¹ · W_in (undo the output projection, redo the input projection), but W_out is non-square so there’s no true inverse. They solve it as ridge regression:

min_{W_a} ‖ W_out · W_a − W_in ‖²_F + λ ‖W_a‖²_F, closed-form W_a = (W_outᵀ W_out + λI)⁻¹ W_outᵀ W_in

Plain English: “find the linear map that best turns output-space vectors into input-space vectors, with a small penalty to keep it stable.” It’s a d_h × d_h matrix (1024² for the small model), computed once and reused for every step of every run. Negligible cost, no training.

3. Latent working-memory transfer (between agents)

After agent A₁ finishes its m latent steps, you grab its entire KV-cache across all L layers — call it M_{A₁}. Crucially this cache covers both the original input tokens AND the m latent-thought positions, because the KV-cache accumulates a key/value pair for every position the model has processed.

M_{A₁} = { (K^(l)_cache, V^(l)_cache) : l = 1..L }, where each cache spans positions 1 .. t+m.

To hand off to agent A₂, you prepend A₁’s per-layer K and V tensors onto A₂’s own (empty or input) KV-cache, layer by layer. Now when A₂ runs attention, every layer can attend back over A₁’s full internal trace. A₂’s latent thoughts are therefore conditioned on A₁’s complete reasoning — no text ever written, no re-encoding.

Theorem 3.3 (Lossless transfer): A₂’s outputs after receiving M_{A₁} equal its outputs if A₁’s real outputs had been fed in directly.

Why it’s lossless: attention only ever reads K and V. If you supply the exact K/V that the previous content would have produced, attention can’t tell the difference. And transferring K/V rather than the raw hidden states avoids making A₂ recompute the projections.

This chains: A₃ inherits M_{A₂} (which already folds in A₁), and so on. Only the final agent decodes to text.

Architecture & data flow

flowchart LR
  Q[Question q] --> A1
  subgraph A1[Agent 1 · Planner]
    direction TB
    A1in[Input embeds E] --> A1lat[m latent steps:\nh_t -> Wa -> e_t+1 -> forward]
    A1lat --> A1kv[KV-cache all L layers\nM_A1 = input + latent]
  end
  A1kv -->|prepend K,V layer-wise| A2
  subgraph A2[Agent 2 · Solver]
    direction TB
    A2lat[latent steps conditioned\non M_A1 + own context] --> A2kv[M_A2 = M_A1 + own]
  end
  A2kv -->|prepend| A3[Agent 3 ...]
  A3 --> FIN[Final agent:\nDECODE to text answer a]
  FIN --> ANS([Answer])

Schematic: a single latent step packs `d_h/log|V|` times more information than one decoded token. Drag the model size; watch how many text tokens one latent step replaces, and how the token budget collapses. Illustrative, built from the paper's formula, not raw logs.

The algorithm, simplified

# LatentMAS: agents think and talk purely in latent space.
# Stubs: forward(emb, kv) -> (hidden_states, new_kv);  ridge(...) -> matrix
# Shapes: h, e are [d_h];  KV per layer is (K,V) each [seq, d_h]

def precompute_alignment(W_out, W_in, lam=1e-2):
    # Maps output-space hidden states back into input-embedding space.
    # Closed-form ridge regression: W_a = (W_outᵀW_out + λI)⁻¹ W_outᵀ W_in
    return ridge(W_out, W_in, lam)            # one d_h×d_h matrix, reused forever

def agent_latent_pass(emb_seq, incoming_kv, W_a, m=40):
    # emb_seq: this agent's input embeddings (question + role prompt)
    # incoming_kv: predecessor's full layer-wise KV-cache (or None for first agent)
    kv = prepend(incoming_kv)                  # transplant predecessor's K,V per layer
    hidden, kv = forward(emb_seq, kv)          # prefill; kv now holds input positions
    h = hidden[-1]                             # last-layer hidden state of last position
    for _ in range(m):                         # think for m steps WITHOUT decoding
        e = h @ W_a                            # realign output -> valid input embedding
        hidden, kv = forward(e, kv)            # feed thought straight back in
        h = hidden[-1]                         # next latent thought
    return kv                                  # M_Ai = input context + m latent thoughts

def latent_mas(question, agents, W_a):
    kv = None
    for agent in agents[:-1]:                  # all but the last collaborate in latent space
        emb = embed(question, agent.role_prompt)
        kv = agent_latent_pass(emb, kv, W_a)   # lossless handoff via KV-cache
    # only the final agent decodes to text
    emb = embed(question, agents[-1].role_prompt)
    return decode_to_text(emb, kv)             # the single text output of the whole system

Built on Prior Work

Results & Evidence

Setup: Qwen3 at 4B/8B/14B, off-the-shelf, no fine-tuning. 9 benchmarks: math/science (GSM8K, AIME24/25, GPQA-Diamond, MedQA), commonsense (ARC-Easy/Challenge), code (MBPP+, HumanEval+). Two MAS topologies (sequential planner-critic-refiner-solver; hierarchical experts + summarizer). Baselines: Single model, Sequential TextMAS, Hierarchical TextMAS. m ∈ {0,10,20,40,80} latent steps; 3 runs averaged; 8× A100.

Headline numbers:

• Accuracy: beats single-model by avg +14.6% (sequential) / +13.3% (hierarchical); beats text MAS by +2.8% / +4.6%. So it’s not just “MAS beats single model” — latent beats text MAS too.
• Tokens: 70.8% / 83.7% fewer output tokens than text MAS; even 15-60% fewer than a single agent (because the final agent mostly aggregates latent memory and decodes a short answer).
• Speed: ~4x / 4.3x faster end-to-end, and still 2.6x-7x faster than text MAS even after the text baselines are accelerated with vLLM (prefix caching + tensor parallel). That last point matters — they didn’t win by leaving the baseline un-optimized.
• Latent step depth: accuracy climbs and peaks around 40-80 steps, then plateaus/slightly declines (redundant thoughts). On AIME, <50 latent steps matches text MAS that needs 20k+ tokens.
• W_a ablation: +2.3-5.3% accuracy; visualizations show unaligned h_t drifts off the input-embedding manifold and W_a pulls it back.
• Semantics check: the latent-thought embeddings occupy nearly the same region as — and actually cover more of — the embedding space of the correct text tokens (on 300 MedQA questions). Evidence that latent thoughts are meaningful, not noise.

What the evidence does NOT establish (read this before you over-buy):

• Single model family. Everything is Qwen3. No Llama, no Mistral, no closed models. KV-cache transplant assumes all agents share the same tokenizer, architecture, and weights — this is homogeneous MAS, not heterogeneous (you can’t latently pass from a Qwen agent to a Llama agent).
• The theory leans on assumptions. Theorem 3.1’s huge “470x” multiplier rests on the Linear Representation Hypothesis and a lossless-encodability premise. The empirical lift (+2.8-4.6%) is real but far smaller than the theoretical expressiveness ratio — so don’t quote 470x to a client as a real-world speedup; the honest number is ~4x.
• No interpretability of the protocol. You can’t read what agents “said.” The case study (Appendix D) asks a model to interpret its own latent thoughts, but there’s no audit trail like text gives you — a real cost for regulated/debuggable systems.
• Topologies are fixed and simple. Sequential chains and one-level hierarchies. No loops, no dynamic routing, no tool use mid-latent.

How You’d Use It

For an AI services shop, the appeal is cost and latency on multi-agent products you already ship in text. Where it slots in:

• Self-hosted / open-weight deployments. This only works when you control the model internals (hidden states + KV-cache). If you’re calling OpenAI/Anthropic APIs, you cannot do this — there’s no API surface for hidden states. So it’s a play for clients running Qwen/Llama on their own GPUs (on-prem, VPC, regulated industries). That’s actually a nice positioning wedge: “we make your private MAS 4x cheaper to run.”
• High-volume internal reasoning pipelines where the intermediate agent chatter is never shown to a user anyway (a planner→solver→checker pipeline that only surfaces the final answer). You lose nothing by hiding the text, and you cut the token bill ~80%.
• Latency-sensitive MAS. Anything where end-to-end response time is the product (live assistants, agentic search) — 4x faster is felt by users.

Where it does not fit: anything needing an auditable reasoning trace (compliance, legal, medical sign-off), heterogeneous agents (different vendors/models), or pure API-only stacks. And the “thoughts are uninterpretable” property is a genuine liability you must disclose.

Realistic framing for a proposal: this is a 20-40% inference-cost reduction with a small accuracy bump on self-hosted homogeneous MAS — concrete, defensible, not magic.

Build Your Own (Minimal Recipe)

You can prototype the 80% version on one open model in a weekend using HuggingFace Transformers (the paper uses the past_key_values interface directly).

Components, in build order:

1. Latent loop for one agent. Take a Qwen3/Llama in HF, run a forward pass with output_hidden_states=True and use_cache=True, grab the last hidden state, feed it back as inputs_embeds for the next step. Loop m times. This is the riskiest-feeling part but is ~30 lines. (Hard part #1: getting the inputs_embeds + past_key_values plumbing right so the cache grows correctly.)
2. Compute W_a once. Pull model.get_input_embeddings().weight (W_in) and the LM head weight (W_out), solve the closed-form ridge regression W_a = (W_outᵀW_out + λI)⁻¹ W_outᵀ W_in with a single torch.linalg.solve. Apply e = h @ W_a before each feedback step. (This is what makes it actually work vs. degenerate — don’t skip it.)
3. KV transfer between two agents. After agent 1’s loop, read past_key_values (a tuple of per-layer (K,V) tensors), and prepend them along the sequence dimension into agent 2’s past_key_values before its forward pass. (Hard part #2: tensor shapes — [batch, n_heads, seq, head_dim] — and making sure positional encodings/RoPE stay consistent after prepending.)
4. Wrap it as a chain. Roles via system prompts (planner, solver), final agent calls normal .generate() to decode text.

Reach for: HuggingFace Transformers (past_key_values, inputs_embeds), a single open model you have weights for (Qwen3-4B is the cheapest faithful repro), vLLM later for the speed numbers. The authors’ code is at github.com/Gen-Verse/LatentMAS — start there to avoid the RoPE/shape footguns.

How to Improve It

1. Heterogeneous latent bridges. The biggest limitation is same-model-only. Train a small adapter (a learned W_bridge) that maps agent A’s hidden space into agent B’s input space when they’re different models. This turns LatentMAS from a homogeneous trick into a real cross-vendor protocol — high value, clearly testable (does a Qwen→Llama latent handoff beat text handoff?).
2. Learned latent collaboration (the authors’ own future-work hook). Everything here is training-free. Apply RL post-training (GRPO-style) where the reward is final-answer correctness and the policy controls how many latent steps each agent takes and what it attends to in the transferred cache. Could push the accuracy lift well past +4%.
3. Adaptive latent depth. They use fixed m. Add a tiny learned “halting” head (à la PonderNet) that stops latent thinking when the hidden state stabilizes — saves the redundant-thought decline seen past 80 steps and adapts compute to question hardness.
4. Partial-interpretability decode. Periodically decode a summary token-stream from the latent thoughts (without conditioning on it) purely as an audit log. You keep the speed of latent collaboration but regain a debuggable trace — directly attacks the “uninterpretable” liability that blocks regulated deployments.
5. Latent + tools. Right now latent collaboration is pure reasoning. Define a protocol for an agent to break out of latent mode, call a tool, and re-enter latent mode with the tool result encoded as KV. That’s the bridge to production agentic systems.

Glossary

• MAS (multi-agent system) — several LLM agents with roles (planner, solver, etc.) coordinating to answer one question.
• Latent / hidden state (h) — the dense d_h-dimensional vector a transformer layer produces for a position; the model’s internal representation before it’s projected to words.
• W_in / W_out — the input embedding matrix (token id → vector) and the LM head (final hidden state → vocab logits).
• KV-cache — stored Key and Value tensors per layer per position; lets attention avoid recomputing earlier positions. Here it doubles as transferable “working memory.”
• Autoregressive — generating one step at a time, each conditioned on all previous; here the “step” is a latent vector, not a token.
• d_h (hidden dimension) — width of the model’s vectors (1024 for Qwen3-0.6B, larger for bigger models); drives the expressiveness ratio.
• |V| (vocabulary size) — number of distinct tokens; log|V| ≈ bits of info per token.
• Linear Representation Hypothesis — the assumption that semantic concepts correspond to linear directions in hidden space; makes the expressiveness bound provable.
• Ridge regression — least-squares fit with an L2 penalty (λ) for stability; used to compute the alignment matrix W_a in closed form.
• Representation drift (OOD activations) — when fed-back hidden states fall outside the distribution the model expects at its input, degrading quality over steps; W_a corrects it.
• Training-free — uses pretrained weights as-is; no gradient updates, only a one-time closed-form matrix solve.
• vLLM — a high-throughput inference engine (prefix caching, tensor parallelism); used to make the baselines fast so the speedup claim is fair.
• GRPO — a reinforcement-learning method (group-relative policy optimization) that reinforces better outputs relative to a sampled group; a candidate for the future “learned” version.