Research map / Large language models / Retrieval and knowledge
Accelerating Iterative Retrieval-augmented Language Model Serving with Speculation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RaLMSpec, which speeds up iterative retrieval-augmented LM serving via speculative retrieval from a local cache, batched verification with rollback, plus prefetching, an optimal speculation stride scheduler (OS3), and asynchronous verification, while preserving outputs.
Problem
Iterative retrieval-augmented language models incur high serving latency because retrieval is issued sequentially and frequently during generation.
Approach
Replace expensive per-step retrieval with speculative retrieval from a request-level local cache exploiting temporal/spatial locality, then verify several speculative steps at once with a batched knowledge-base retrieval and roll back on mismatch, so outputs are provably unchanged.
Key results
- With exact dense retriever, RaLMSpec+PSA reduces latency by up to 2.39x (GPT2), 2.33x (OPT), 1.75x (LLaMA-2-7B) vs. RaLMSeq.
- With approximate dense retriever, speed-up is only 1.04-1.39x; for KNN-LM up to 2.45x.
- For KNN-LM with exact dense retriever, speed-up up to 7.59x with OS3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add speculative retrieval and batched verification | Iterative RaLM serving · QA workloads with EDR | serving latency: 1.75×-2.39× speed-up | Helped |
| Add prefetching, OS³, and asynchronous verification (ablation) | RaLMSpec · Iterative RaLM across EDR, ADR, and SR | speed-up ratio: 2.39×; 2.32×; 1.75× (EDR); 1.05×; 1.39×; 1.04× (ADR); 1.53×; 1.77×; 1.31× (SR) | Helped |
| Add the optimal speculation stride scheduler (ablation) | RaLMSpec · Approximate dense and sparse retrievers | serving latency | Helped |
| Increase prefetch size from 20 to 256 (ablation) | RaLMSpec with prefetching · Iterative RaLM workloads | serving latency | Hurt |
| Add asynchronous verification (ablation) | RaLMSpec · Iterative RaLM workloads | serving latency | Mixed |
| Use fixed speculation stride 4 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, EDR | serving latency: 85.19s -> 81.06s | Helped |
| Use fixed speculation stride 2 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, ADR | serving latency: 8.14s -> 9.86s | Hurt |
| Use fixed speculation stride 4 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, ADR | serving latency: 8.14s -> 14.93s | Hurt |
| Use fixed speculation stride 8 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, ADR | serving latency: 8.14s -> 25.88s | Hurt |
| Use fixed speculation stride 2 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, SR | serving latency: 10.38s -> 10.65s | Hurt |
| Use fixed speculation stride 4 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, SR | serving latency: 10.38s -> 12.48s | Hurt |
| Use fixed speculation stride 8 instead of OS³ (ablation) | RaLMSpec · LLaMA-2-7B, Wiki QA, SR | serving latency: 10.38s -> 16.66s | Hurt |
Surprising findings
- With approximate dense or sparse retrievers, the naive fixed-stride RaLMSpec is sometimes slower than the baseline (e.g., 0.62x with ADR on GPT2).
- Prefetching with size 256 often reduces performance due to retrieval overhead.
Limitations
- Speed-up bottlenecked by retrieval share of end-to-end latency; marginal gains with approximate dense retrievers and large LMs.
- Asynchronous verification latency is simulated due to Python GIL.
- Fixed stride can degrade performance; OS3 needs warm-up phase.
Experimental setup
Datasets: Wiki-QA Web Questions Natural Questions TriviaQA WikiText-103 Wikipedia corpus
Benchmarks: Wiki-QA Web Questions Natural Questions TriviaQA WikiText-103
Metrics: speed-up ratio serving latency (seconds)
Baselines: RaLMSeq (naive iterative RaLM serving) KNN-LM serving baseline
Builds on: KNN-LM Dense Passage Retriever BM25 FAISS
Closest papers by meaning
- Improving Retrieval Augmented Language Model with Self-Reasoning (AAAI 2025 · similarity 0.88)
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning (ICLR 2024 · similarity 0.87)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.86)
- Scaling Retrieval-Based Language Models with a Trillion-Token Datastore (NeurIPS 2024 · similarity 0.86)
- LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference (ICLR 2026 · similarity 0.86)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.