Research map / Architectures and efficiency / RNNs and normalization
Distribution-Driven Dense Retrieval: Modeling Many-to-One Query-Document Relationship
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DDR, which represents queries as vectors and documents as diagonal Gaussian distributions, scoring relevance by Gaussian log-density rewritten as a dot product so standard ANN indexes apply.
Problem
Dense retrieval must handle many-to-one query-document relationships; multi-vector document representations incur increased storage, vector collapse under sparse labels, and high search latency.
Approach
Model each document as a multivariate Gaussian (mean from [CLS], diagonal variance via attention pooling) and each query as a vector; the log-likelihood of the query under the document Gaussian is algebraically converted into a dot product of query features [1, q, q^2] and document features, enabling ANN search with a single vector per document.
Key results
- On MS MARCO dev, DDR (multi-vector comparison setting) reaches MRR@10 0.402 and R@50 0.874 with 52GB index, versus ColBERTv2 0.397 at 154GB and CITADEL 0.399 at 81GB.
- DDR with 383-dim projection (single-vector storage) reaches MRR@10 0.396 on MS MARCO dev, exceeding MRL (0.393) and other single-vector baselines.
- Zero-shot NDCG@10: DDR scores 0.514 on NQ, 0.307 on FiQA and 0.315 on Touché-2020, above CITADEL on those three datasets, though it is lower than CITADEL on TREC DL 2020 (0.698 vs 0.702).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using single-vector retrieval | Dense retrieval · MS MARCO DEV | MRR@10: 0.399 -> 0.402 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using single-vector retrieval | Dense retrieval · MS MARCO DEV | R@50: 0.862 -> 0.874 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using single-vector retrieval | Dense retrieval · MS MARCO DEV | R@1k: 0.981 -> 0.984 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using multi-vector retrieval | Dense retrieval · TREC-DL '19 | NDCG@10: 0.738 -> 0.731 | Hurt |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using multi-vector retrieval | Dense retrieval · TREC-DL '20 | NDCG@10: 0.702 -> 0.698 | Hurt |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using the strongest listed baseline | Dense retrieval · NQ | NDCG@10: 0.510 -> 0.514 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using the strongest listed baseline | Dense retrieval · FiQA | NDCG@10: 0.302 -> 0.307 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using the strongest listed baseline | Dense retrieval · Touché-2020 | NDCG@10: 0.294 -> 0.315 | Helped |
| represent documents as Gaussian distributions and queries as vectors (DDR) rather than using the strongest listed baseline | Dense retrieval · Climate-FEVER | NDCG@10: 0.228 -> 0.193 | Hurt |
| vary β from 0.1 to 10 when smoothing the document variance vector (ablation) | DDR · MS MARCO | performance | No clear effect |
Surprising findings
- DDR is lower than CITADEL on TREC DL 2020 (0.698 vs 0.702) and does not beat BM25-level gains by much on zero-shot Climate-FEVER (0.193 vs CITADEL 0.191 is only marginal), and the authors attribute TREC DL discrepancies to small test sets rather than a real gap.
Limitations
- Document distributions are restricted to diagonal Gaussians, which limits covariance modeling.
- TREC DL evaluation uses very few queries (43 and 54), so differences are noisy; the authors acknowledge about 2% seed variation.
- Gains over strong baselines such as CITADEL are small on several zero-shot datasets.
Experimental setup
Datasets: MS MARCO TREC DL 2019 TREC DL 2020 NQ FiQA Touché-2020 Climate-FEVER
Benchmarks: MS MARCO passage dev TREC DL 2019 TREC DL 2020 BEIR NQ BEIR FiQA BEIR Touché-2020 BEIR Climate-FEVER
Metrics: MRR@10 R@50 R@1k NDCG@10 retrieval latency (ms/query) storage (GB)
Baselines: BM25 DeepCT SPLADE DPR ANCE ADORE RocketQA RocketQAv2 Margin-MSE TAS-B CLDRD MRL
Builds on: TAS-B MRL (multivariate representation learning) word Gaussian embeddings ColBERTv2 MVR
Closest papers by meaning
- Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.85)
- Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (AAAI 2024 · similarity 0.85)
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.84)
- TriSampler: A Better Negative Sampling Principle for Dense Retrieval (AAAI 2024 · similarity 0.84)
- Fine-Grained Distillation for Long Document Retrieval (AAAI 2024 · similarity 0.84)
- Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.