Research map / Architectures and efficiency / RNNs and normalization
A Gradient Accumulation Method for Dense Retriever under Memory Constraint
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ContAccum, a gradient-accumulation-based memory reduction method for training BERT dense retrievers with InfoNCE under 11GB memory, using dual (query and passage) memory banks of past representations to enlarge the negative set and stabilize training.
Problem
Training dense retrievers with InfoNCE needs large batches for many in-batch negatives, which requires costly multi-accelerator hardware; existing memory reduction methods (gradient accumulation, GradCache, pre-batch negatives) are either reduce negatives, slow, or unstable.
Approach
Maintain FIFO memory banks for both query and passage representations (stop-gradient, no stored activations) so each local batch's similarity matrix includes many more negatives; gradient analysis shows a passage-only bank causes a gradient norm imbalance between the two encoders, which a dual bank resolves.
Key results
- Under 11GB VRAM, ContAccum reaches NQ Top@20 80.1 versus 79.4 for DPR at 80GB (batch 128), surpassing high-resource DPR.
- ContAccum outperforms high-resource DPR in 18 of 24 metrics, with gains up to 4.9 points.
- A passage-only memory bank (pre-batch-style) drops about 8 Top@20 points on NQ versus ContAccum; GradNormRatio grows up to 30x without the query bank.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use dual query and passage memory banks instead of omitting the query memory bank (ablation) | ContAccum · NQ | Top@20: 70.8 -> 78.8 | Helped |
| use dual query and passage memory banks instead of omitting the query memory bank (ablation) | ContAccum without hard negatives · NQ without hard negatives | Top@20: 72.3 -> 76.3 | Helped |
| use past encoder representations (ablation) | ContAccum · NQ | Top@20: 76.5 -> 78.8 | Helped |
| use past encoder representations (ablation) | ContAccum without hard negatives · NQ without hard negatives | Top@20: 73.4 -> 76.3 | Helped |
| use both the query memory bank and past encoder representations (ablation) | ContAccum · NQ | Top@20: 67.8 -> 78.8 | Helped |
| use both the query memory bank and past encoder representations (ablation) | ContAccum without hard negatives · NQ without hard negatives | Top@20: 73.9 -> 76.3 | Helped |
| use GradAccum (ablation) | ContAccum · NQ | Top@20: 76.7 -> 78.8 | Helped |
| use GradAccum (ablation) | ContAccum without hard negatives · NQ without hard negatives | Top@20: 74.1 -> 76.3 | Helped |
| use ContAccum rather than GradCache under 11GB | BERT dense retriever · MS Marco | NDCG@20: 34.9 -> 39.1 | Helped |
| use ContAccum rather than GradCache under 11GB | BERT dense retriever · MS Marco | Recall@100: 22.4 -> 23.8 | Helped |
| use ContAccum rather than GradCache under 11GB | BERT dense retriever · NQ test | Top@20: 79.5 -> 80.1 | Helped |
| use ContAccum rather than GradCache under 11GB | BERT dense retriever · TriviaQA test | Top@20: 79.4 -> 79.8 | Helped |
Surprising findings
- Past encoder representations were found to be useful from the early stages of training, contrary to prior claims that memory banks should be used only in late epochs.
- Using the memory bank alone without gradient accumulation already improves performance over DPR at the same local batch size.
Limitations
- Evaluated only in supervised fine-tuning; pre-training stage not studied.
- Still relies on softmax over the similarity matrix, which is computationally costly.
- Experiments limited to BERT-base dense retrievers.
Experimental setup
Datasets: Natural Questions TriviaQA Curated TREC WebQuestions MS MARCO
Benchmarks: Natural Questions TriviaQA Curated TREC WebQuestions MS MARCO
Metrics: Top@k NDCG@k Recall@k
Baselines: DPR GradAccum GradCache pre-batch negatives
Builds on: DPR GradCache pre-batch negatives gradient accumulation nano-DPR
Closest papers by meaning
- A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks (ICML 2024 · similarity 0.85)
- Mitigating the Impact of False Negative in Dense Retrieval with Contrastive Confidence Regularization (AAAI 2024 · similarity 0.84)
- Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.84)
- TriSampler: A Better Negative Sampling Principle for Dense Retrieval (AAAI 2024 · similarity 0.84)
- RazorAttention: Efficient KV Cache Compression Through Retrieval Heads (ICLR 2025 · similarity 0.84)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.