Research map / Large language models / Retrieval and knowledge

Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection

ICLR 2025 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Sparse RAG, which pre-fills KV caches of all retrieved documents in parallel and then uses LoRA-tuned Gemini models to rate each context with control tokens, decoding only over the kept contexts.

Problem

RAG input length grows linearly with the number of retrieved documents, so prefill and especially per-token decoding latency become high; existing fixes (FiD, PCW) are either encoder-decoder-specific or only speed up prefill, and filtering methods need extra classifier calls.

Approach

Encode retrieved documents independently in parallel (PCW-style block attention with re-used positions), have the same LLM score each cached context with a control token and "Good" token, and decode only attending to contexts whose score exceeds a threshold, so relevance filtering and generation share one forward pass.

Key results

• On PopQA, Sparse RAG reaches 67.17 EM / 71.16 F1 using on average 7.84 of 20 contexts, versus 65.43 EM / 69.99 F1 for dense RAG, with decoding speed 12.28 t/s vs 6.65 t/s.
• On QMSum, Sparse RAG achieves 23.96 F1 and 20.10 RougeLSum with 4.45 contexts and decoding speed 16.05 t/s, versus 21.43 F1 and 18.20 RougeLSum for dense RAG at 6.64 t/s.
• Sparse RAG's encoding speed matches PCW RAG (147.58 t/s vs 56.28 t/s for dense RAG), and the Gemini-XS variant reaches 68.26 EM on PopQA with 6.27 contexts.

Measured effects reported in the paper

Surprising findings

• Filtering contexts improves quality over using all 20 documents, with the best PopQA F1 at a threshold of 0.15 and continued decline at higher thresholds.
• Using LLM-generated relevance labels from a two-round PaLM2 and Gemini pipeline outperformed silver title-matching labels used by Corrective RAG.

Limitations

• Requires additional tuning on top of the LLM, so it is not plug-and-play.
• When per-context assessment labels are missing, the LLM rating prompts may need adjustment for each use case.
• Relies on proprietary Gemini models that are not open-source, limiting reproducibility; statistical significance not reported.

Experimental setup

Datasets: PopQA QMSum Natural Questions

Benchmarks: PopQA QMSum

Metrics: Exact Match F1 RougeLSum Encoding Speed (tokens/s) Decoding Speed (tokens/s)

Baselines: RAG PCW RAG Corrective RAG

Builds on: Parallel Context Windows Fusion-in-Decoder LoRA Corrective RAG

Closest papers by meaning

• Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.92)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.92)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.91)
• RAGGED: Towards Informed Design of Scalable and Stable RAG Systems (ICML 2025 · similarity 0.91)
• Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting (ICLR 2025 · similarity 0.90)
• Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems (ICLR 2025 · similarity 0.89)