Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RMSC, which jointly trains source human labels and target weak supervision for dense retrieval, using learned domain and relevance soft tokens to decouple data domain and supervision type for zero-shot retrieval.
Problem
Zero-shot dense retrieval degrades out of domain because DR models trained on source labels overfit source patterns, and mixing target pseudo-labels with source labels without distinguishing them is unstable.
Approach
Use special soft tokens (domain tokens and relevance tokens) prepended/inserted into inputs so the DR model knows which domain and supervision type each training example comes from; apply the same weak-supervision extraction to source corpus so only domain and relevance differ, then jointly train and infer with target-domain and human-label tokens.
Key results
- On LoTTE, RMSC(CO) averages 0.366 NDCG@10 vs 0.336 for the best baseline (COCO-DR at 0.327 for COCO and 0.336 for Condenser-based rows as reported), with ~14% gain over best ANCE on NDCG@10 per the text.
- On BEIR, RMSC(CO) averages 0.497 NDCG@10 vs 0.486 for GenQ-style/Contriever baseline ranking, outperforming COCO-DR (0.476) with statistical significance on several datasets.
- Training curves show RMSC keeps out-of-domain BEIR performance steadily improving, whereas direct fine-tuning of Condenser and COCO-DR oscillates or degrades.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RMSC to COCO-DR initialization | COCO-DR · Lotte Avg | NDCG@10: 0.336 -> 0.366 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · Lotte Avg | Recall@100: 0.566 -> 0.642 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · Lotte Avg | Recall@1000: 0.762 -> 0.828 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · BEIR Avg. | NDCG@10: 0.476 -> 0.497 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · TREC-COVID | NDCG@10: 0.743 -> 0.772 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · SciFact | NDCG@10: 0.734 -> 0.742 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · NFCorpus | NDCG@10: 0.357 -> 0.355 | Helped |
| Apply RMSC to COCO-DR initialization | COCO-DR · BEIR datasets | NDCG@10 | Mixed |
| Jointly train with target-domain weak supervision and source human labels using domain and relevance soft tokens | RMSC · BEIR out-of-domain datasets during training | NDCG@10 | Helped |
Surprising findings
- Direct fine-tuning of Condenser and COCO-DR shows out-of-domain performance peaking and then declining while in-domain MS MARCO keeps improving, indicating source overfitting.
Limitations
- Uses only random negative sampling, so it does not exploit hard-negative or distillation techniques that could further improve results.
- Weak supervision is limited to positive pairs from SCL, ICT or QGen; the relevance gap between weak and human labels is only mitigated, not closed.
Experimental setup
Datasets: MS MARCO LoTTE BEIR
Benchmarks: LoTTE BEIR
Metrics: NDCG@10 Recall@100 Recall@1000
Baselines: BM25 DPR ANCE TAS-B Condenser Contriever COCO-DR GenQ GPL
Builds on: Condenser COCO-DR Contriever QGen GPL
Closest papers by meaning
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.86)
- Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.85)
- Distribution-Driven Dense Retrieval: Modeling Many-to-One Query-Document Relationship (AAAI 2025 · similarity 0.85)
- USTAD: Unified Single-model Training Achieving Diverse Scores for Information Retrieval (ICML 2024 · similarity 0.85)
- A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks (ICML 2024 · similarity 0.84)
- Evaluating Transferability in Retrieval Tasks: An Approach Using MMD and Kernel Methods (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.