Research map / Label-efficient and robust learning / Recognition and long-tail
Generative Retrieval Meets Multi-Graded Relevance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GR² (GRaded Generative Retrieval), which generates relevant and distinct docids via regularized fusion of a docT5query docid generator and an autoencoder, and trains T5-base with a multi-graded constrained contrastive loss, in supervised and Wikipedia-pretraining variants.
Problem
Generative retrieval has been limited to binary relevance; extending it to multi-graded relevance is hard because docid likelihoods are not comparable across lengths and multiple relevant documents may share similar or identical docids.
Approach
Jointly fuse a query-generation latent space with an autoencoder latent space (relevance and distinctness regularizers) to produce distinct pseudo-query docids, and train with a contrastive loss whose pull strength is scaled by relevance grade, with a grade-wise max constraint across grades.
Key results
- On Gov 500K, GR^2P reaches nDCG@5 0.5095 and P@20 0.5506, versus RIPOR's 0.4713 and 0.4831 (about 11% relative P@20 gain per the paper).
- On MS 500K, GR^2P achieves MRR@20 0.3835 and Hits@1 0.3821, outperforming RIPOR (MRR@20 0.3626, Hits@1 0.3421).
- On MS 1M, GR^2P is comparable to RIPOR (MRR@20 0.3659 vs 0.3674).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GR²P instead of RIPOR | T5-base generative retrieval · Gov 500K | P@20: 0.4831 -> 0.5506 | Helped |
| use GR²P instead of RIPOR | T5-base generative retrieval · ClueWeb 500K | nDCG@20: 0.2707 -> 0.2969 | Helped |
| use GR²P instead of RIPOR | T5-base generative retrieval · Robust04 | nDCG@20: 0.3849 -> 0.4205 | Helped |
| use GR²P instead of RIPOR | T5-base generative retrieval · MS 500K | MRR@20: 0.3626 -> 0.3835 | Helped |
| use GR²P instead of RIPOR | T5-base generative retrieval · NQ 320K | MRR@20: 0.4173 -> 0.4418 | Helped |
| pre-train on constructed multi-graded Wikipedia data before fine-tuning | GR² · Gov 500K, ClueWeb 500K, and Robust04 | retrieval performance | Helped |
| remove regularized fusion and use pseudo-queries from a single QG model as docids (ablation) | GR² · Gov 500K and MS 500K | retrieval performance | Hurt |
| add regularized fusion docids instead of using the MLE variants' docid setup (ablation) | GR² · Gov 500K and MS 500K | retrieval performance | Helped |
| remove the grade penalty λ (ablation) | GR² MGCC · Gov 500K | retrieval performance | Hurt |
| remove the grade constraint LMax (ablation) | GR² MGCC · Gov 500K | retrieval performance | Hurt |
| remove the grade penalty λ or grade constraint LMax (ablation) | GR²S MGCC · MS 500K | retrieval performance | No clear effect |
| replace MGCC with weighted cross-entropy (ablation) | GR² · Gov 500K and MS 500K | retrieval performance | Hurt |
Surprising findings
- Existing GR baselines perform worse than the QLM sparse baseline on Gov 500K and Robust04.
- In the zero-resource setting all GR methods underperform BM25; GR^2P only beats BM25 on ClueWeb 500K nDCG@20 in the low-resource setting.
Limitations
- Docids are fixed after initialization and not jointly optimized with retrieval.
- Uses hard weights per relevance grade; soft assignment not explored.
- Still lags behind full-ranking BM25+monoBERT on Gov 500K and MS 500K.
Experimental setup
Datasets: Gov2 ClueWeb09-B Robust04 MS MARCO Natural Questions English Wikipedia MS 1M
Benchmarks: Gov 500K ClueWeb 500K Robust04 MS 500K NQ 320K MS 1M
Metrics: nDCG@5 nDCG@20 P@20 ERR@20 MRR@3 MRR@20 Hits@1 Hits@10
Baselines: BM25 DocT5query QLM SPLADE RepBERT DPR PseudoQ ANCE DSI-Num DSI-Sem DSI-QG NCI
Builds on: DSI NCI RIPOR docT5query supervised contrastive loss LambdaRank
Closest papers by meaning
- DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (AAAI 2025 · similarity 0.91)
- Descriptive and Discriminative Document Identifiers for Generative Retrieval (AAAI 2025 · similarity 0.91)
- Learning to Rank in Generative Retrieval (AAAI 2024 · similarity 0.89)
- UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · similarity 0.88)
- GENIUS: A Generative Framework for Universal Multimodal Search (CVPR 2025 · similarity 0.87)
- Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.