Research map / Large language models / Retrieval and knowledge

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs

NeurIPS 2024 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 30 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Instruction-tunes a single LLM (Llama3-8B/70B) to both rerank retrieved contexts and generate RAG answers, using a small amount of ranking data mixed into the instruction-tuning blend.

Problem

Standard retrieve-then-generate RAG pipelines rely on weak dense retrievers and separate expert rerankers that generalize poorly, and LLMs degrade when given many retrieved contexts, so high-recall top-k selection remains difficult.

Approach

Instruction-tune one LLM for both context ranking (framed as True/False relevance QA and retrieval-augmented multi-passage ranking) and answer generation, so the LLM reranks top-N retrieved passages to top-k before generating; the unified (question, context, answer) format lets tasks transfer to each other.

Key results

• Llama3-RankRAG 70B averages 56.1 across nine general-domain benchmarks vs 53.6 for Llama3-ChatQA-1.5 70B and 42.0 for GPT-4-0613.
• Llama3-RankRAG 8B averages 52.6 vs 49.6 for Llama3-ChatQA-1.5 8B, with over 10% gains on PopQA and 2WikimQA.
• On the Mirage biomedical RAG benchmark, Llama3-RankRAG 70B reaches 78.06 average, about 98% of GPT-4-0613 (79.97), without biomedical fine-tuning.

Measured effects reported in the paper

Surprising findings

• Directly using ChatQA-1.5 to rank passages hurts performance, while adding a small amount of ranking data to instruction tuning makes the LLM a better reranker than dedicated rerankers trained on about 10x more ranking data.
• Ranking with RankRAG enables k=5 to work well, unlike vanilla RAG where performance trends differ with k.

Limitations

• Focuses on single-round retrieval; combining with multi-round RAG is left for future work.
• Reranking requires one LLM forward pass per retrieved passage, adding latency that grows with N.
• Ranking data for conversational settings is pseudo-labeled from 4-gram recall, which may be noisy.

Experimental setup

Datasets: OpenAssistant Dolly SODA ELI5 Self-Instruct Unnatural Instructions FLAN Chain-of-thought datasets DROP NarrativeQA Quoref ROPES

Benchmarks: NQ TriviaQA PopQA HotpotQA 2WikimQA FEVER Doc2Dial TopiOCQA INSCIT Mirage

Metrics: Exact Match Accuracy F1 Recall@5/10/20

Baselines: ChatQA-1.5 Llama3-Instruct GPT-4-0613 GPT-4-turbo-2024-0409 GPT-3.5-turbo-1106 RankLLaMA BGE-Reranker-v2-m3 monoT5

Builds on: ChatQA RAFT RankLLaMA

Closest papers by meaning

• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.92)
• Cooperative Retrieval-Augmented Generation for Question Answering: Mutual Information Exchange and Ranking by Contrasting Layers (NeurIPS 2025 · similarity 0.90)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models (AAAI 2025 · similarity 0.90)
• Retrieval meets Long Context Large Language Models (ICLR 2024 · similarity 0.90)
• DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.90)
• Provence: efficient and robust context pruning for retrieval-augmented generation (ICLR 2025 · similarity 0.89)