Research map / Architectures and efficiency / RNNs and normalization
Semi-Parametric Retrieval via Binary Bag-of-Tokens Index
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SiDR, a semi-parametric bi-encoder that aligns a learned parametric query/document representation with a fixed binary bag-of-tokens index, enabling BM25-cost indexing with neural-level effectiveness, plus a late parametric re-ranking mode.
Problem
Neural retrievers require costly embedding of the whole corpus and index rebuilds whenever parameters change, which hurts online indexing, low-cost deployment, and co-training retrievers with LLMs; term-based BM25 is cheap but less effective.
Approach
A disentangled bi-encoder (building on VDR) trained with a semi-parametric contrastive loss that aligns the parametric query/passage vocabulary-space representations with a non-parametric binary bag-of-tokens (BoT) document index, so documents can be indexed via tokenization alone; a late parametric mode first retrieves with the BoT index and re-ranks top-m with the neural encoder.
Key results
- On Wiki21m NQ, the non-parametric SiDR-beta reaches 39.8 top-1 accuracy versus 22.7 for BM25, and the parametric SiDR reaches 49.1 vs 46.0 for DPR.
- The late parametric SiDR-beta (m=100) reaches 50.3 top-1 on NQ, surpassing the primary baselines, while embedding only about 1% of the 21M passages per query set.
- On BEIR, the non-parametric SiDR-beta averages 32.5 NDCG@10 versus 41.1 for BM25, with better results than BM25 only on in-domain MS MARCO (19.0 vs 18.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SiDR's parametric index instead of DPR | SiDR full · NQ test (Wiki21m) | top-1 accuracy: 46.0 -> 49.1 | Helped |
| Use SiDR's BoT index instead of BM25 | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 22.7 -> 39.8 | Helped |
| Use SiDR beta with an index-only pipeline instead of BM25 | SiDR beta · BEIR out-of-domain datasets | NDCG@10 | Mixed |
| Rerank the top 100 candidates with SiDR | SiDR beta · BEIR average | NDCG@10: 32.5 -> 43.4 | Helped |
| Add in-training retrieved negatives (ablation) | SiDR full · NQ test (Wiki21m) | top-1 accuracy: 44.9 -> 49.1 | Helped |
| Add in-training retrieved negatives (ablation) | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 32.3 -> 39.8 | Helped |
| Add negative samples (ablation) | SiDR full · NQ test (Wiki21m) | top-1 accuracy: 30.2 -> 49.1 | Helped |
| Add negative samples (ablation) | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 24.4 -> 39.8 | Helped |
| Add the semi-parametric objective (ablation) | SiDR full · NQ test (Wiki21m) | top-1 accuracy: 43.8 -> 49.1 | Helped |
| Add the semi-parametric objective (ablation) | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 12.3 -> 39.8 | Helped |
| Set the retrieved-negative pool to m=1 rather than the baseline m=20 (ablation) | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 39.8 -> 41.2 | Helped |
| Set the retrieved-negative pool to m=100 rather than the baseline m=20 (ablation) | SiDR beta · NQ test (Wiki21m) | top-1 accuracy: 39.8 -> 37.3 | Hurt |
Surprising findings
- Removing the semi-parametric loss drops SiDR-beta top-1 accuracy from 39.8 to 12.3 on NQ, while the parametric model is much less affected.
- The non-parametric BoT index underperforms BM25 on most out-of-domain BEIR datasets, despite beating it in-domain on Wikipedia QA.
Limitations
- Non-parametric index is more sensitive to distribution shift and underperforms BM25 on most out-of-domain BEIR datasets.
- In-training retrieval relies on answer strings to identify negatives, so it is only used for Wikipedia QA benchmarks, not MS MARCO.
- Results are reported with a single BERT-base-scale encoder; compared baselines are only from moderate-cost settings, with advanced baselines not integrated.
Experimental setup
Datasets: Natural Questions TriviaQA WebQuestions MS MARCO Wikipedia 21M passages
Benchmarks: Wiki21m (NQ, TriviaQA, WebQuestions) MS MARCO BEIR (12 datasets)
Metrics: top-1/5/20 retrieval accuracy MRR@10 NDCG@10 indexing and search latency
Baselines: DPR BM25 VDR SPLADE uniCOIL ANCE E5-base Contriever
Builds on: VDR DPR BM25 MLM pre-training
Closest papers by meaning
- Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoders (ICLR 2024 · similarity 0.86)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.85)
- Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe (NeurIPS 2025 · similarity 0.85)
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.84)
- Retrieval-based Disentangled Representation Learning with Natural Language Supervision (ICLR 2024 · similarity 0.84)
- BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.