Research map / Image generation / Autoregressive and latent generation
Learning to Rank in Generative Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LTRGR, which adds a learning-to-rank training phase with a margin-based rank loss on top of a generative retrieval model (MINDER) so that passage ranking is optimized end-to-end, without changing inference.
Problem
Generative retrieval models learn to generate passage identifiers as an intermediate goal, and a heuristic function converts predicted identifiers into a ranked passage list; this disconnect creates a learning gap with the passage ranking objective.
Approach
After standard generative training, the autoregressive model is further trained with a margin-based rank loss over the passage rank list produced by the identifier-to-passage scoring heuristic, combined with the generation loss, so the heuristic is inside the training loop.
Key results
- On NQ, LTRGR reaches hits@5 of 68.8 vs. 65.8 for MINDER, and on MS MARCO it improves MINDER's Recall@5 by 10.7 points and MRR@10 by 6.9 points (to 25.5).
- On TriviaQA, LTRGR reaches hits@100 of 85.1, but trails DPR at hits@5 (70.2 vs 72.7) and hits@20 (79.1 vs 80.2).
- Applying the same learning-to-rank phase to SEAL improves its NQ hits@5 from 61.3 to 63.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the learning-to-rank training phase to MINDER | MINDER (BART-Large) · MSMARCO | Recall@5: 29.5 -> 40.2; 10.7 points | Helped |
| Add the learning-to-rank training phase to MINDER | MINDER (BART-Large) · MSMARCO | MRR@10: 18.6 -> 25.5; 6.9 points | Helped |
| Add the learning-to-rank training phase to MINDER | MINDER (BART-Large) · Natural Questions | hits@5: 65.8 -> 68.8; 3.0 | Helped |
| Add the learning-to-rank training phase to MINDER | MINDER (BART-Large) · TriviaQA | hits@5: 68.4 -> 70.2; 1.8 | Helped |
| Add the learning-to-rank training phase to SEAL | SEAL · Natural Questions | hits@5: 61.3 -> 63.7; 3.6 points | Helped |
| Add the learning-to-rank training phase to SEAL | SEAL · Natural Questions | hits@100: 86.3 -> 86.4; 0.1 points | Helped |
| Remove both margin-based rank losses (ablation) | LTRGR · Natural Questions | hits@5: 68.8 -> 65.8 | No clear effect |
| Remove generation loss (ablation) | LTRGR · Natural Questions | hits@5: 68.8 -> 63.9 | Hurt |
| Remove rank loss 1 (ablation) | LTRGR · Natural Questions | hits@5: 68.8 -> 68.2 | Hurt |
| Remove rank loss 2 (ablation) | LTRGR · Natural Questions | hits@5: 68.8 -> 67.9 | Hurt |
| Replace margin-based rank loss with list-wise infoNCE loss (ablation) | LTRGR · Natural Questions | hits@5: 68.8 -> 65.4 | Hurt |
| Set generation-loss weight λ to 0 (ablation) | LTRGR · Natural Questions | hits@5: 65.7 | Hurt |
Surprising findings
- Training with only the rank loss (no generation loss) underperforms the original MINDER, as the model falls into local minima and assigns small scores to all passages.
- A list-wise infoNCE rank loss performed worse than the margin-based loss (NQ hits@5 65.4 vs 68.8).
Limitations
- Gains on TriviaQA do not surpass DPR at hits@5 and hits@20.
- Generative retrieval cannot see passage content, and suffers from autoregressive error accumulation.
- Hyperparameters (margin 500, lambda 1000) were tuned on the NQ test set, and the list-wise loss was only trained for one epoch.
Experimental setup
Datasets: Natural Questions TriviaQA MS MARCO
Benchmarks: Natural Questions TriviaQA MS MARCO
Metrics: hits@5 hits@20 hits@100 Recall@5 Recall@20 Recall@100 MRR@10
Baselines: BM25 DPR GAR DSI DSI (scaling up) NCI SEAL MINDER
Builds on: MINDER SEAL DSI FM-index
Closest papers by meaning
- DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval (AAAI 2025 · similarity 0.90)
- Generative Retrieval Meets Multi-Graded Relevance (NeurIPS 2024 · similarity 0.89)
- Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.89)
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (NeurIPS 2024 · similarity 0.87)
- UniGen: A Unified Generative Framework for Retrieval and Question Answering with Large Language Models (AAAI 2024 · similarity 0.87)
- Sparse Meets Dense: Unified Generative Recommendations with Cascaded Sparse-Dense Representations (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.