Research map / Label-efficient and robust learning / Recognition and long-tail
Descriptive and Discriminative Document Identifiers for Generative Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learnable descriptive and discriminative n-gram DocIDs (D2-DocID), selected with a corpus-aware NR-IDR score, and the D2Gen generative retrieval model built on T5-base.
Problem
Generative document retrieval relies on DocIDs that are chosen only for relevance to their document, ignoring whether they distinguish the document from similar documents in the corpus.
Approach
Select n-gram DocIDs using a corpus-level document-ngram attention matrix with an NR-IDR score (TF-IDF-inspired, combining relevance, term frequency, and inverse document relevance), then iteratively retrain the generative retriever with contrastive-assisted understanding and diversity-filtered synthetic queries.
Key results
- On MS300k, D2Gen reaches M@10 0.615 and R@1 0.513, outperforming the best baseline (MINDER/TSGen/NOVO-level) by about 4.5% on R@1 as claimed.
- On NQ320k, D2Gen achieves the best R@1 (0.710) among compared methods, with M@10 0.772 comparable to TSGen (0.771) and NOVO (0.767).
- Swapping in D2-DocID improves SE-DSI (M@10 0.469 to 0.536) and Ultron (0.400 to 0.538) on MS300k, showing the DocID design transfers across models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use D2Gen with descriptive and discriminative DocIDs | Generative retrieval · MS300k | M@10: 0.592 -> 0.615* | Helped |
| Use D2Gen with descriptive and discriminative DocIDs | Generative retrieval · NQ320k | R@1: 0.708 -> 0.710 | Helped |
| Replace Ultron's DocID with D2-DocID | Ultron · MS300k | M@10: 0.400 -> 0.538 | Helped |
| Replace SE-DSI's DocID with D2-DocID | SE-DSI · MS300k | M@10: 0.469 -> 0.536 | Helped |
| Replace D2Gen's DocID with D2-DocID | D2Gen · MS300k | M@10: 0.579 -> 0.607 | Helped |
| Use 2 iterations instead of 1 (ablation) | D2Gen · MS300k | M@10: 0.603 -> 0.610 | Helped |
| Add contrastive loss Lc to the retrieval pipeline (ablation) | D2Gen · MS300k | M@10: 0.592 -> 0.607 | Helped |
| Use 3 n-grams instead of 2 in the DocID (ablation) | D2Gen · MS300k | M@10: 0.594 -> 0.607 | Helped |
| Use 3 n-grams instead of 6 in the DocID (ablation) | D2Gen · MS300k | M@10: 0.603 -> 0.607 | Helped |
| Use 3 n-grams instead of 9 in the DocID (ablation) | D2Gen · MS300k | M@10: 0.580 -> 0.607 | Helped |
| Set the similarity threshold to 0.99 instead of 0.9 (ablation) | D2Gen query filtering · MS300k | M@10: 0.607 -> 0.616 | Helped |
| Use NR-IDR instead of TF-IDF for DocID selection (ablation) | D2Gen · MS300k | M@10: 0.586 -> 0.607 | Helped |
Surprising findings
- On NQ320k, D2Gen's R@10 (0.876) and R@100 (0.936) are lower than several baselines such as NOVO and GenRet, so gains are not uniform across metrics.
- Choosing more n-grams per DocID (9) hurts retrieval, and too strict a similarity threshold (0.99 vs 0.9 in ablation) also changes results, indicating sensitivity to DocID length and augmentation filtering.
Limitations
- Gains are smaller on NQ320k, where structured titles already summarize documents well.
- Recall at larger cutoffs (R@10, R@100) lags some baselines on NQ320k.
- Iterative DocID selection and retraining adds pipeline complexity; the paper does not report its compute cost.
Experimental setup
Datasets: MS MARCO Natural Questions
Benchmarks: MS300k NQ320k
Metrics: MRR@10 MRR@100 Recall@1 Recall@10 Recall@100
Baselines: BM25 UniCOIL SPLADEv2 DPR ANCE GTR-Base DSI NCI GENRE Ultron SEAL MINDER
Builds on: DSI SE-DSI MINDER NOVO TF-IDF
Closest papers by meaning
- DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (AAAI 2025 · similarity 0.91)
- Generative Retrieval Meets Multi-Graded Relevance (NeurIPS 2024 · similarity 0.91)
- Language Models as Semantic Indexers (ICML 2024 · similarity 0.86)
- Learning to Rank in Generative Retrieval (AAAI 2024 · similarity 0.85)
- Information-Theoretic Generative Clustering of Documents (AAAI 2025 · similarity 0.85)
- Retrieval is Accurate Generation (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.