Research map / Label-efficient and robust learning / Recognition and long-tail
Contextual Document Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes contextual document embeddings via an adversarial (cluster-based) contrastive batch construction and a two-stage Contextual Document Encoding architecture that conditions each document embedding on embeddings of neighboring corpus documents.
Problem
Dense document embeddings are computed independently per document, so they cannot adapt to corpus-specific statistics (like IDF) and degrade under domain shift at test time in retrieval.
Approach
Two complementary methods: (1) Adversarial Contrastive Learning that clusters training data into hard, self-similar batches so in-batch negatives are challenging; (2) a Contextual Document Encoding architecture where a second-stage transformer conditions on first-stage embeddings of neighboring corpus documents, so document embeddings depend on the test corpus context.
Key results
- Adversarial batch construction improves biencoder training at both batch size 256 and 16,384 across domains.
- The contextual CDE architecture trained with small batch size 256 outperforms biencoder baselines without large batches or supervised data.
- Largest gains occur on severely out-of-domain datasets such as TREC-Covid and FiQA; combining both methods gives the best performance on most domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adversarially selected batches instead of in-domain random batches | biencoder training · BEIR test datasets | NDCG@10 | Helped |
| use a small batch of 256 with CDE instead of relying on large training batches | CDE · BEIR test datasets | NDCG@10 | Helped |
| add contextual document encoding | biencoder · BEIR test datasets | NDCG@10 | Helped |
| combine adversarially chosen batches with contextual document encoding | retrieval model · BEIR test datasets | NDCG@10 | Helped |
| use out-of-domain context instead of in-domain context (ablation) | CDE · BEIR test datasets (aggregate) | NDCG@10: 0.499 -> 0.466 | Hurt |
| use random in-domain context instead of top-k context for both document and query encoding (ablation) | CDE · BEIR test datasets (aggregate) | NDCG@10 | Helped |
| provide context documents from the task's own domain instead of another domain (ablation) | CDE · CQADupstack | NDCG@10 | Helped |
| increase the number of contextual examples toward a full context window (ablation) | CDE · BEIR test datasets | NDCG@10 | Helped |
| remove context by using null context (ablation) | CDE · BEIR test datasets | NDCG@10 | Hurt |
| train with large random in-domain batches instead of small adversarially selected batches (ablation) | biencoder · BEIR test datasets | NDCG@10 | Helped |
Surprising findings
- Replacing document context with out-of-domain 'fake' text only moderately reduces performance (0.499 to 0.466), and random in-domain context gives larger aggregate gains than top-k nearest neighbors.
Limitations
- Experiments restricted to a single type of dual-encoder retrieval model that encodes documents into single vectors.
- Mainly English datasets; multilingual settings left to future work.
- No error bars or statistical significance reported.
Experimental setup
Datasets: Nomic meta-datasets (unsupervised and supervised, incl. HotpotQA and MS MARCO)
Benchmarks: BEIR (twelve datasets incl. TREC-Covid, FiQA, Quora, CQADupStack)
Metrics: NDCG@10
Baselines: biencoder baselines with batch sizes 256 and 16,384 BM25 GTR Contriever E5 GTE Nomic-Embed
Builds on: GTR NomicBERT GradCache BEIR FAISS
Closest papers by meaning
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.88)
- Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT (ICML 2024 · similarity 0.87)
- Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.87)
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.87)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.87)
- In-Context Pretraining: Language Modeling Beyond Document Boundaries (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.