MoR: Better Handling Diverse Queries with a Mixture of Sparse, Dense, and Human Retrievers

TL;DR

RAG quality lives or dies on retrieval, and no single retriever wins on every query — BM25 nails keyword matches, dense models nail paraphrases, and which one wins flips query by query. Today people just fix one retriever by gut feel. This paper proposes Mixture-of-Retrievers (MoR): send every query to all retrievers, compute a per-query, per-retriever trustworthiness weight using cheap signals derived from embedding geometry (no training, no labels), then take a weighted average of their relevance scores and re-rank. A mixture of eight BERT-sized retrievers (0.8B params total) beats every individual component by +10.8% and beats 7B LLM retrievers like GritLM by +3.9% on average. The same weighting machinery even works on “human retrievers” (noisy domain experts), boosting collaborative performance +58.9% over the humans alone.

Problem & Motivation

The pain in one sentence: the retriever you hard-code into your RAG pipeline is wrong for a large fraction of your queries, and you have no principled, label-free way to know which retriever to trust for any given query.

Concretely: on encyclopedic questions, dense retrievers (DPR) beat BM25. On medicine and biology, the lexical BM25 baseline stays strong. The paper makes this vivid with a win-rate analysis on SciFact — even though TAS-B beats DPR overall, DPR still “wins” (finds the gold document when TAS-B doesn’t) on 11.7% of queries. SimCSE and DPR have nearly identical aggregate scores yet disagree on ~40% of queries. So there’s real, exploitable, query-level complementarity sitting on the table.

The standard fixes fall short:

• Pick one retriever by heuristic → leaves the 11–40% of “off-distribution” queries unserved.
• Train a router/classifier → needs labels, needs retraining when you add a retriever, and brittle across domains.
• Naive fusion (e.g., Reciprocal Rank Fusion) → treats all retrievers as equally trustworthy per query, so a confidently-wrong retriever drags the others down.

The authors’ framing: this is the old distributed / aggregated search and query performance prediction (QPP) problem from classical IR, reborn inside RAG. They want a zero-shot weighting function that, for each query, estimates “how much should I believe retriever R right now?” — from geometry alone.

What’s New (Core Contribution)

1. Per-query, zero-shot retriever weighting from embedding geometry.
   • Before: you fix a retriever, or learn a router with labeled data.
   • Now: a label-free function f(q, Rᵢ, D) scores each retriever’s trustworthiness for this query using only distances in the retriever’s own embedding space. No training, plug-and-play.
2. Multi-granularity “deep fusion” to expand the retriever pool for free.
   • Before: a retriever indexes documents at one granularity (whole passages).
   • Now: each retriever is replicated across 4 index granularities (questions↔documents, questions↔propositions, sub-questions↔passages, sub-questions↔propositions), turning N retrievers into 4N candidates with zero model changes.
3. Pre- + post-retrieval signal design that beats ad-hoc fusion.
   • Before: fusion = mean/max of scores, or RRF, treating retrievers as equal.
   • Now: a pre-retrieval familiarity signal (query-to-corpus-cluster distance) usable before you even run the search, plus post-retrieval signals (Moran coefficient + document-to-corpus distance) that adapt the older QPP literature into the weighting.
4. “Human as a retriever.” Because the weighting only needs ranked outputs + an embedding model, you can drop a noisy human expert into the mixture and MoR learns to up-weight them in their domain and ignore them elsewhere — quantifying human trustworthiness from the corpus alone.

Honest read on novelty: the individual ingredients (MoE intuition, QPP, Moran coefficient, propositional indexing) are borrowed. The genuinely new move is wiring QPP-style signals into a zero-shot per-query mixture weight and showing small-model mixtures beat big-model retrievers. The “human retriever” result is a clever stress test but uses simulated oracle experts, so treat it as a proof-of-concept, not a deployed system.

How It Works (Technically)

The whole method is a weighted sum. For a query q and document dⱼ, each retriever Rᵢ already gives a relevance score sᵢ(q, dⱼ) (e.g., cosine similarity), normalized to [0,1]. MoR computes an adjusted score:

$$\tilde{s}(q, d_j) = \sum_{i=1}^{N} f(q, R_i, D), s_i(q, d_j)$$

In plain English: re-score every document as a trust-weighted average of what all retrievers think of it, then re-rank by the new score. All the intelligence lives in f — the per-query, per-retriever weight. Everything else is bookkeeping.

So the entire paper reduces to: how do you compute f(q, Rᵢ, D) without labels? Three signals.

1. Pre-retrieval signal V_pre — “does this query look like the stuff this retriever knows?”

This is computed before searching. The idea borrows from the when-to-retrieve literature (model familiarity): a retriever is trustworthy on a query if the query embedding sits comfortably inside the dense regions of that retriever’s corpus. Operationally:

• Run KMeans on the corpus D in Rᵢ’s embedding space → clusters C₁..C_K with centroids m⃗ₖ (they set K = max(ceil(⁴√|D|), 3)).
• For the query vector q⃗, look at the vectors to each centroid: v⃗ₖ = m⃗ₖ − q⃗.

$$V_{pre}(q, R_i, D) \triangleq \sum_{k=1}^{K} \frac{|C_k|}{K} \cdot \hat{v}_k \cdot \frac{1}{|\vec{v}_k|_2^2}$$

Decoding it term by term:

• 1 / ‖v⃗ₖ‖² — inverse-square distance. Close clusters dominate; far clusters barely contribute. If the query is near a cluster, that term is huge.
• |Cₖ| / K — cluster size weight. Big (dominant) clusters count more, so being near a popular region matters more than being near a tiny niche.
• v̂ₖ — the unit direction to the centroid. Summing directions matters: if the query is pulled equally toward many scattered clusters, the directions partly cancel and the signal stays low (the query is “ambiguous” to this retriever). If it’s decisively near one region, the vectors align and the magnitude is large.
• The result is a vector; its magnitude is the scalar trust score. Large = “this query clearly belongs to a dominant region of my corpus, trust me.” Small = outlier or ambiguous → down-weight this retriever.

The clever part: you get a trust estimate before paying for retrieval, which later enables an efficiency trick (reject low-V_pre retrievers entirely).

2. Post-retrieval signal — Moran coefficient I_Moran — “are the documents I retrieved mutually consistent?”

After retrieving, take the top-k results. The Moran coefficient (a spatial-autocorrelation measure from classical QPP, Diaz 2007) asks: do the retrieved documents cluster together in embedding space? This rests on the cluster hypothesis — relevant documents resemble each other. If the top results are tightly bunched, the retriever found a coherent neighborhood (probably relevant); if they’re scattered, it was guessing. Higher Moran → more trust.

3. Post-retrieval signal V_post — “do the retrieved docs themselves look like the corpus core?”

This reuses V_pre, but applied to the retrieved documents instead of the query. For the top-20 retrieved docs D_q:

$$V_{post}(q, R_i, D) \triangleq \frac{1}{|D_q|} \sum_{n=1}^{|D_q|} V_{pre}(d_n, R_i, D)$$

In words: average how “central” each retrieved document is to the corpus. If a retriever surfaced documents that sit in dense, well-understood regions, trust it more.

Combining the signals into f:

• MoR-pre (cheapest): f = V_pre only. No retrieval needed to weight.
• MoR-post (best): a fixed linear blend f = a·V_pre + b·I_Moran + c·V_post with hand-tuned (a, b, c) = (0.1, 0.3, 0.6), the same coefficients for every query. (They flag query-specific coefficients as future work.)

The deep-fusion expansion (orthogonal to weighting): before any of this, each retriever is cloned across 4 granularities using a “propositioner” (Chen et al. 2023b) that breaks queries into sub-questions and documents into atomic propositions (e.g., “Alice and Bob had coffee” → “Alice had coffee” + “Bob had coffee”). A keyword retriever might match sub-questions to propositions better than full passages. This 4× expansion needs no model changes — it just gives the mixture more, finer-grained signals to weight.

Architecture & data flow

flowchart TB
  Q[Query q] --> EXP
  subgraph EXP[Deep fusion: each retriever x4 granularities]
    R1[BM25 variants]
    R2[DPR variants]
    R3[SimCSE / Contriever / TAS-B / GTR / MPNet ...]
  end
  EXP --> SCORES["Each retriever scores every doc: s_i q,d_j"]

  Q --> WPRE["Pre-retrieval signal V_pre: query vs corpus cluster centroids"]
  SCORES --> WPOST["Post-retrieval signals: Moran coeff + V_post over top-20 docs"]
  WPRE --> F["Weight f q,R_i,D = a*V_pre + b*Moran + c*V_post"]
  WPOST --> F

  F --> AGG["Weighted sum: s_tilde = sum_i f * s_i q,d_j"]
  SCORES --> AGG
  AGG --> RR[Re-rank documents by adjusted score]
  RR --> OUT[Top-k docs to the reader LLM]

Interactive: a query (white dot) over two retrievers' corpus clusters. Drag the query around; watch each retriever's V_pre trust weight change with proximity to its dense clusters, and watch the final weighted ranking shift. This is the core MoR intuition — geometry decides trust.

The algorithm, simplified

# MoR core: zero-shot, per-query weighted fusion of many retrievers.
# Stubs: retriever.score(q) -> dict{doc_id: relevance in [0,1]}
#        retriever.embed(x) -> vector;  retriever.clusters -> precomputed KMeans on corpus

def v_pre(vec, clusters):
    # "How decisively does this vector belong to a dominant corpus region?"
    total = None
    for c in clusters:                       # c.centroid, c.size, K total clusters
        diff = c.centroid - vec              # v_k : query -> centroid
        dist2 = (diff @ diff)                # squared L2 distance
        term = (c.size / len(clusters)) * (diff / (dist2 ** 0.5)) * (1.0 / dist2)
        total = term if total is None else total + term   # sum of weighted unit directions
    return norm(total)                       # magnitude = trust scalar

def mor_post(q, retrievers, top_k=20, a=0.1, b=0.3, c=0.6):
    adjusted = defaultdict(float)
    for R in retrievers:                     # each is a (model x granularity) variant
        scores = R.score(q)                  # {doc: s_i(q, doc)}, normalized [0,1]
        top = sorted(scores, key=scores.get, reverse=True)[:top_k]

        pre   = v_pre(R.embed(q), R.clusters)                    # before-retrieval familiarity
        moran = moran_coefficient([R.embed(d) for d in top])    # do top docs cluster together?
        post  = mean(v_pre(R.embed(d), R.clusters) for d in top) # are top docs corpus-central?

        weight = a * pre + b * moran + c * post  # trust THIS retriever for THIS query
        for doc, s in scores.items():
            adjusted[doc] += weight * s          # weighted-sum fusion (the one equation)
    return sorted(adjusted, key=adjusted.get, reverse=True)      # re-rank

That’s the whole contribution: no gradients, no training loop. The only “learning” is offline KMeans on the corpus and the three hand-set coefficients.

Built on Prior Work

Results & Evidence

Setup. Four scientific-domain retrieval benchmarks chosen for diverse/complex queries (not encyclopedic): NFCorpus (medical), SciDocs (multi-domain papers), SciFact (claim verification), SciQ (science exam QA). Metric: NDCG@5 / NDCG@20 (ranking quality). Pool: 8 sparse+dense retrievers totaling 0.836B params. Reference baselines: RepLLaMA-7B and GritLM-7B.

Headline numbers:

• MoR-post beats its own best components by +10.8% (vs best unsupervised) and +12.2% (vs best supervised) relative NDCG@20.
• MoR-post (0.8B total) beats GritLM-7B by +3.9% relative NDCG@20 on average — wins on NFCorpus, SciFact, SciQ; comparable on SciDocs.
• Route Oracle (an upper-bound where you magically pick the single best retriever per query) beats GritLM by +13.5% — confirming the headroom MoR is chasing.
• RAG end-to-end (Exact Match, reader = Llama-3-8B): MoR-post tops baselines on SciFact (72.9 EM@1 vs GritLM 66.9) and is competitive on SciQ — gains survive into generation.
• Human-retriever stress test: with 4 simulated domain experts (oracle in-domain, random out-of-domain), V_post assigns ~0.6–0.8 weight to each expert in their own domain and ~0.0 elsewhere (Table 5 — clean diagonal). MoR+Humans hits 87–94 NDCG@20 per domain vs ~40–71 for humans alone: +58.9% relative.
• Efficiency: mixing the best 2 retrievers ≈ mixing all 8 on SciQ (92.6 vs 92.9). And the best pair is not the two best individual models — complementarity beats raw strength. Pre-rejecting low-V_pre retrievers at the 95th percentile keeps most performance using only ~20% of retrievers per query.
• Ablation: the designed signals (50.7–58.7 avg NDCG@20) crush naive mean (35.2) and max (46.3) fusion; deep-fusion granularity merging adds real lift.

What the evidence does NOT establish (read this before selling it):

• Narrow domain. Everything is scientific/technical retrieval with deliberately complex queries. No web-scale, conversational, or multi-hop agentic benchmarks. Gains may shrink where one retriever already dominates.
• The “human retriever” win is simulated. Experts are oracle in-domain by construction — that’s an idealized ceiling, not evidence real noisy annotators behave this way.
• Hand-tuned coefficients (0.1, 0.3, 0.6) are global, picked empirically, with no held-out tuning protocol shown — mild risk of test-set leakage on so few datasets.
• Cost honesty: “0.8B beats 7B” counts parameters, but MoR runs many encoders + KMeans + per-query post-retrieval scoring. Wall-clock/latency comparison vs a single GritLM call isn’t directly given; the efficiency section is about trimming the pool, not beating a single big model on latency.

How You’d Use It

For an AI-services shop building RAG for clients, this is a drop-in retrieval upgrade that needs no fine-tuning and no labeled data — which is exactly the constraint most client engagements hit.

• Heterogeneous-corpus clients (legal + medical + internal wiki). A single embedding model underperforms on at least one slice. Stand up BM25 + 2–3 dense models, let MoR weight per query. You get domain-adaptive retrieval without training a domain model — a clean differentiator versus “we used OpenAI embeddings.”
• Agentic retrieval tool. In a multi-agent system, the retrieval tool is usually one retriever. Swap in MoR so the tool itself is robust to whatever weird query an agent throws at it (keyword lookups, paraphrased questions, multi-condition claims). The pre-retrieval signal also gives the agent a cheap confidence score — “no retriever is confident here” is a useful signal to trigger a different action (ask user, decompose, web search).
• Human-in-the-loop knowledge work. The “human as retriever” framing maps directly onto consulting reality: a domain SME hands you a ranked shortlist. MoR can blend the SME’s picks with automated retrieval and learn from the corpus how much to trust them per query — defensible, explainable weighting you can show a client.
• Efficiency lever for proposals. “We get 7B-retriever quality from 0.8B of small models, and we can trim to the best 2 per domain” is a real cost/latency story for budget-conscious clients.

Build Your Own (Minimal Recipe)

You can get ~80% of the value with MoR-pre + a 2-retriever pool in an afternoon.

Components & build order:

1. Pick 2–3 complementary retrievers — one lexical (BM25 via rank_bm25 or Elasticsearch), one+ dense (sentence-transformers: e.g., all-mpnet-base-v2, contriever). Complementarity > raw strength.
2. Pre-compute corpus embeddings per dense retriever (offline; this is where the time goes).
3. KMeans the corpus per retriever (sklearn.cluster.KMeans, K = max(ceil(|D|^0.25), 3)). Cache centroids + sizes.
4. Implement v_pre exactly as the pseudocode above — it’s ~10 lines of numpy.
5. At query time: score docs with each retriever, compute weight = v_pre(query) per retriever, take the weighted-sum of normalized scores, re-rank. That’s MoR-pre.
6. Upgrade to MoR-post by adding the Moran coefficient + V_post over the top-20 (needs pysal/esda for Moran, or implement spatial autocorrelation directly).

The 1–2 genuinely hard parts:

• Score normalization & calibration. Cosine, BM25, and dot-product scores live on different scales; the weighted sum is meaningless unless you normalize each retriever’s scores to [0,1] consistently (min-max per query is the paper’s approach). Get this wrong and the strongest-scoring retriever silently dominates.
• The deep-fusion proposition step (optional). Running a propositioner to atomize queries/docs is the most engineering-heavy piece. Skip it for v1 — the weighting alone delivers most of the lift; add granularity expansion later.

Reach for: sentence-transformers, rank_bm25, scikit-learn (KMeans), faiss (ANN search at scale), numpy. No GPU training, no RL, no labeled data.

How to Improve It

1. Learn the coefficients (the obvious win). Replace the fixed (0.1, 0.3, 0.6) with a tiny MLP that maps query embedding → per-query, per-retriever (a, b, c). The paper explicitly flags this; with even a few hundred labeled queries you could fit it and likely close part of the gap to Route Oracle (which still beats MoR-post by a lot — that 13.5% over GritLM vs MoR’s 3.9% is unclaimed headroom).
2. Add post-presentation signals. MoR stops at retrieval. Feed back downstream signals — did the answer pass a verifier? did EM improve? — to re-weight retrievers online. This turns MoR into a bandit over retrievers and connects it to agentic retrieval where the reward is task success.
3. Complementarity-aware pool selection. The efficiency result shows the best pair isn’t the two best models. Formalize this: compute pairwise query-win disagreement and greedily select a maximally complementary subset, rather than enumerating all subsets. Could yield a principled “which 3 retrievers” recipe per corpus.
4. Real human-retriever deployment. Replace simulated oracle experts with actual annotators producing noisy rankings, and test whether V_post still recovers their trustworthiness. This is where a genuine human-LLM-collaboration product would be validated (or fall apart).
5. Latency-honest evaluation + caching. Publish wall-clock vs GritLM, and explore caching V_pre per query-cluster so the per-query overhead amortizes. Pair with the 95th-percentile early-rejection to make MoR genuinely cheaper, not just smaller-in-params.
6. Combine with cross-encoder re-ranking. MoR produces a strong candidate set cheaply; a single cross-encoder pass on its top-50 could add precision the bi-encoder mixture can’t reach — a natural two-stage pipeline.

Glossary

• RAG (Retrieval-Augmented Generation) — feed an LLM retrieved documents at inference so it answers from sources instead of memory; reduces hallucination.
• Retriever — a model/function that, given a query, scores/ranks documents in a corpus by relevance.
• Sparse retriever (BM25) — ranks by lexical token overlap (TF-IDF style); great for exact keyword matches, no neural net.
• Dense retriever (DPR, SimCSE, Contriever, etc.) — encodes query and docs into vectors; ranks by vector similarity (cosine/dot); captures meaning/paraphrase.
• NDCG@K — Normalized Discounted Cumulative Gain over the top-K results; standard ranking-quality metric, higher is better, rewards putting relevant docs near the top.
• Exact Match (EM@K) — fraction of generated answers that exactly match the gold answer, using K retrieved chunks.
• Mixture-of-Experts (MoE) — architecture where a gating network routes each input to specialized sub-models; MoR borrows the idea, treating each retriever as an expert.
• Query Performance Prediction (QPP) — classical IR task: estimate how good a retrieval result is without relevance labels.
• Moran coefficient — a spatial-autocorrelation statistic; here, measures whether the top retrieved documents cluster together in embedding space (a proxy for “this retrieval is coherent/relevant”).
• Cluster hypothesis — the IR assumption that documents relevant to the same query tend to be similar to each other.
• Centroid — the mean vector of a cluster of points; KMeans represents each cluster by its centroid.
• KMeans — unsupervised clustering that partitions points into K groups around centroids; used here offline on the corpus per retriever.
• Proposition / sub-question (deep fusion) — atomic decomposition of a document into single-fact sentences (or a query into sub-questions) to create finer-grained index units.
• Zero-shot — works without task-specific training or labeled data; MoR’s weighting is zero-shot because it’s pure geometry.
• Route Oracle — an upper-bound baseline that magically routes each query to its single best retriever; used to measure how much headroom MoR is leaving on the table.
• GritLM / RepLLaMA — 7B-parameter LLM-based retrievers used as strong reference baselines.