Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-attention re-ranking over BM25 top-100 | BM25 · BEIR 18 datasets | average nDCG@10 vs BM25: + 11% | Improved |
| add late interaction (ColBERT) | BM25 · BEIR | average nDCG@10 vs BM25: + 2.5% | Improved |
| add document expansion (docT5query) | BM25 · BEIR | average nDCG@10 vs BM25: + 1.6% | Improved |
| replace BM25 with learned term weighting (DeepCT) | BM25 · BEIR | average nDCG@10 vs BM25: - 27.9% | Hurt |
| replace BM25 with SPARTA | BM25 · BEIR | average nDCG@10 vs BM25: - 20.3% | Hurt |
| replace BM25 with DPR | BM25 · BEIR | average nDCG@10 vs BM25: - 47.7% | Hurt |
| evaluate in-domain MS MARCO vs zero-shot | BM25 · MS MARCO vs BEIR | nDCG@10: 0.228 vs 0.408 (TAS-B) | Hurt |
| use TAS-B (Margin-MSE) instead of ANCE | dense retrievers · BEIR | nDCG@10 | Improved |
| TAS-B instead of ANCE on TREC-COVID | dense retrievers · TREC-COVID | nDCG@10: 0.654 -> 0.481 | Hurt |
| fine-tune TAS-B on synthetic queries (GenQ) | TAS-B · BEIR specialized domains | nDCG@10 | Mixed |
| manually add missing annotations (remove holes) | ANCE · TREC-COVID | nDCG@10: 0.654 -> 0.735 | Improved |
| manually add missing annotations (remove holes) | docT5query · TREC-COVID | nDCG@10: 0.713 -> 0.714 | No clear effect |
Closest papers by meaning
- Semi-Parametric Retrieval via Binary Bag-of-Tokens Index (ICLR 2025 · similarity 0.88)
- Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.88)
- MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encoding (NeurIPS 2024 · similarity 0.87)
- Benchmarking Large Language Models in Retrieval-Augmented Generation (AAAI 2024 · similarity 0.87)
- Self-Retrieval: End-to-End Information Retrieval with One Large Language Model (NeurIPS 2024 · similarity 0.87)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
