Research map / Label-efficient and robust learning / Recognition and long-tail
Mitigating the Impact of False Negative in Dense Retrieval with Contrastive Confidence Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a contrastive confidence regularizer for NCE loss and a model-agnostic "passage sieve" that filters false-negative hard negatives, improving DPR and AR2 dense retrieval on NQ, TriviaQA and MS MARCO.
Problem
False negatives (unlabeled relevant passages) in dense retrieval training data, which hard negative sampling amplifies, degrade contrastive training of passage retrievers.
Approach
Extends peer-loss/CORES² confidence regularization to the NCE loss by subtracting a beta-weighted mean NCE over all in-batch passages, with a theoretical guarantee; uses the resulting robust model to sieve hard negatives whose loss is below the average loss, producing a cleaner training set for any retriever.
Key results
- AR2+passage sieve reaches 79.2 R@5 on NQ versus 77.9 for AR2 (batch 64, 15 hard negatives), and 78.7 R@5 on TQ versus 78.2.
- DPR+CCR with batch size 128 improves NQ R@20 from 78.4 to 79.5 and TQ R@20 from 79.3 to 79.8 over DPR.
- On NQ, AR2+passage sieve with only 5 hard negatives (78.0 R@5) approaches AR2 with 15 hard negatives (77.9 R@5); the sieve adds roughly one-tenth of AR2 training time in the reported efficiency table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add passage sieve to AR2 | AR2 · NQ test | R@20: 86.0 -> 86.4 | Helped |
| Add passage sieve to AR2 | AR2 · TQ test | R@20: 84.4 -> 84.7 | Helped |
| Add passage sieve to AR2 | AR2 · MS-pas development | MRR@10: 39.5 -> 39.8 | Helped |
| Add passage sieve to AR2 | AR2 · MS-pas development | R@50: 87.8 -> 88.3 | Helped |
| Add passage sieve to AR2 | AR2 · MS-pas development | R@1k: 98.6 -> 98.6 | No clear effect |
| Add contrastive confidence regularizer | DPR · NQ test | R@20: 78.4 -> 79.5 | Helped |
| Add contrastive confidence regularizer | DPR · NQ test | R@100: 85.4 -> 86.1 | Helped |
| Add contrastive confidence regularizer | DPR · TQ test | R@20: 79.3 -> 79.8 | Helped |
| Add contrastive confidence regularizer | DPR · TQ test | R@100: 84.9 -> 85.1 | Helped |
| Use batch size 32 instead of 64 (ablation) | AR2+passage sieve · NQ test | R@20: 86.4 -> 86.1 | Hurt |
| Use batch size 32 instead of 64 (ablation) | AR2+passage sieve · TQ test | R@20: 84.7 -> 84.8 | Helped |
| Use 5 rather than 15 hard negatives (ablation) | AR2+passage sieve · NQ test | R@20: 86.1 -> 85.9 | No clear effect |
Surprising findings
- Passage sieve discards more than half of the hard negatives in many epochs (sieve-out rate near 0.5), and the method is reported as insensitive to beta in the tested range.
Limitations
- Only tested on top of AR2 and DPR; other retrievers not evaluated.
- Theory covers NCE loss generally but experiments are limited to dense passage retrieval.
- Built on first-order CORES² statistics; second-order extension left to future work.
Experimental setup
Datasets: Natural Questions (NQ) TriviaQA MS MARCO passage ranking
Benchmarks: Natural Questions (NQ) TriviaQA MS MARCO passage ranking
Metrics: R@5 R@20 R@100 MRR@10 R@50 R@1k
Baselines: DPR ANCE COIL ME-BERT RocketQA RocketQAv2 RDR PAIR Condenser coCondenser ERNIE-Search MVR
Builds on: CORES² peer loss DPR AR2
Closest papers by meaning
- TriSampler: A Better Negative Sampling Principle for Dense Retrieval (AAAI 2024 · similarity 0.87)
- Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.85)
- QuRe: Query-Relevant Retrieval through Hard Negative Sampling in Composed Image Retrieval (ICML 2025 · similarity 0.85)
- CABINET: Content Relevance-based Noise Reduction for Table Question Answering (ICLR 2024 · similarity 0.84)
- A Gradient Accumulation Method for Dense Retriever under Memory Constraint (NeurIPS 2024 · similarity 0.84)
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.