Research map / Label-efficient and robust learning / Recognition and long-tail
A Fresh Take on Stale Embeddings: Improving Dense Retriever Training with Corrector Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a small corrector MLP that transforms stale cached target embeddings to approximate current target encoder outputs, enabling accurate hard-negative sampling for dense retrieval and RLM training without re-embedding targets.
Problem
Training dual-encoder retrievers with softmax over millions of targets requires cached target embeddings, which become stale as the target encoder trains; keeping them fresh requires costly full re-embedding.
Approach
A small parametric target corrector network (two-layer residual MLP) takes stale cached target embeddings and outputs corrected embeddings every step, trained jointly with a cross-entropy loss between truncated softmaxes, so hard negatives are sampled from up-to-date-like logits with zero re-embedding.
Key results
- On NQ test with GTR-base, the corrector (zero re-embeddings) reaches R@1 50.61 vs 50.30 for exhaustive re-embedding (1.68B re-embeds) and 46.76 for stale.
- On NQ with T5-base, the corrector reaches R@1 36.65 vs 37.34 for exhaustive, with zero re-embeddings.
- In RLM training, the corrector matches exhaustive on NQ (52.3 vs 52.4 EM with GTR) while requiring 0 re-embeddings instead of 1.1B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a target corrector to the stale-buffer training method | GTR-base dense retrieval · NQ Dev | Recall@1: 33.11 -> 35.78 | Helped |
| add a target corrector to the stale-buffer training method | GTR-base dense retrieval · NQ Test | Recall@1: 46.76 -> 50.61 | Helped |
| add a target corrector to the stale-buffer training method | T5-base dense retrieval · NQ Dev | Recall@1: 16.79 -> 23.64 | Helped |
| add a target corrector to the stale-buffer training method | T5-base dense retrieval · NQ Test | Recall@1: 27.65 -> 36.65 | Helped |
| use cross-entropy rather than MSE to train the corrector (ablation) | GTR-base dense retrieval · NQ Dev | Recall@1: 34.98 -> 35.78 | Helped |
| reduce corrector hidden units from 8192 to 4096 (ablation) | GTR-base dense retrieval · NQ Dev | Recall@1: 35.78 -> 35.53 | Hurt |
| reduce corrector hidden units from 8192 to 2048 (ablation) | GTR-base dense retrieval · NQ Dev | Recall@1: 35.78 -> 35.55 | Hurt |
| add a target corrector instead of freezing the retriever | GTR retriever with T5-base reader · NQ validation (RLM) | Exact match accuracy: 48.4 -> 52.3 | Helped |
| add a target corrector instead of freezing the retriever | GTR retriever with T5-base reader · TriviaQA validation (RLM) | Exact match accuracy: 55.1 -> 66.4 | Helped |
| add a target corrector instead of freezing the retriever | GTR retriever with T5-base reader · HotPotQA validation (RLM) | Exact match accuracy: 28.0 -> 36.7 | Helped |
| add uniform negatives to exhaustive re-embedding (ablation) | T5-base dense retrieval · NQ Dev | Recall@1: 24.87 -> 24.70 | No clear effect |
| add uniform negatives to exhaustive re-embedding (ablation) | GTR-base dense retrieval · NQ Dev | Recall@1: 36.53 -> 36.29 | No clear effect |
Surprising findings
- Training with a corrector and no re-embedding slightly outperforms exhaustive re-embedding on GTR-base NQ test Recall@1 (50.61 vs 50.30).
Limitations
- Comparison with exhaustive re-embedding shows small residual gap on T5-base (R@1 36.65 vs 37.34).
- Relies on keeping the full target buffer on accelerator memory; large target sets may need subsampling or CPU-resident buffers.
- Theoretical bounds are loose in practice, relying on Wasserstein/Rademacher terms and stated only as bounds.
Experimental setup
Datasets: Natural Questions MSMARCO TriviaQA NQOpen HotPotQA BEIR ArguAna
Benchmarks: Natural Questions MSMARCO TriviaQA NQOpen HotPotQA ArguAna
Metrics: Recall@1 Recall@5 Recall@10 Recall@20 Recall@100 exact match KL divergence precision@K number of re-embeddings
Baselines: In-batch negatives Stale Dynnibal Stochastic Negative Mining Exhaustive re-embedding Frozen retrieval Two-round training
Builds on: GTR Fusion-in-Decoder perplexity distillation Dynnibal
Closest papers by meaning
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.87)
- SALS: Sparse Attention in Latent Space for KV Cache Compression (NeurIPS 2025 · similarity 0.86)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.85)
- A Gradient Accumulation Method for Dense Retriever under Memory Constraint (NeurIPS 2024 · similarity 0.85)
- Contextual Document Embeddings (ICLR 2025 · similarity 0.84)
- Combining Multiple Supervision for Robust Zero-Shot Dense Retrieval (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.