Research map / Large language models / Reasoning and chain of thought

LightRetriever: A LLM-based Text Retrieval Architecture with Extremely Faster Query Inference

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LightRetriever, an LLM-based hybrid retriever that keeps a full LLM on the document side and reduces query encoding to a cached embedding lookup (dense) plus raw token counts (sparse), achieving >1000x query-encoding speedup with ~95% retained retrieval quality.

Problem

LLM-based dense retrievers use deep LLM query encoders on both sides, so online query encoding is slow and resource-hungry, creating a serving bottleneck for real-time retrieval.

Approach

Break the query/document symmetry: train a full LLM document encoder and a query side that is just averaged cacheable per-token embeddings (a single embedding lookup at serving time), plus a sparse query representation from raw token counts with an LLM-projected learned sparse document side, linearly combined into a hybrid score.

Key results

• LightRetriever-Llama3.1-8B reaches 54.4 nDCG@10 on BEIR vs 56.8 for the full symmetric Llama3.1-8B, and 63.0 vs 67.6 on CMTEB-R.
• On a single A800, query encoding for 65,536 queries drops from 109.4853 s (Full-Llama8b) to 0.0412 s, and QPS rises from 549 to 6999 (about 10x end-to-end).
• Across backbones, LightRetriever retains about 95% of full-symmetric performance on average (typically 1-5 nDCG@10 points lower).

Surprising findings

• Symmetric lightweight retrieval on both sides drops performance by 13.8 BEIR points, and training with only an embedding-bag query encoder drops it by 11.2, showing deep query modeling during training is needed even though serving is a lookup.
• Dense-only lightweight queries fail on compositional constraints (Git case study), where the sparse branch recovers the correct ranking.

Limitations

• Performance drops on reasoning-intensive and code retrieval (BRIGHT, CoIR), and on complex compositional queries where dense-only lightweight encoding fails.
• Only tested on LLM backbones; applicability to non-LLM architectures such as BERT is left to future work.
• Average retention of about 95% means a measurable quality loss relative to full symmetric retrievers.

Experimental setup

Datasets: MSMARCO Amazon Review (2018) Eli5 FEVER FiQA GooAQ HotpotQA NFCorpus NPR NQ PAQ Quora Duplicates

Benchmarks: BEIR CMTEB Retrieval CoIR BRIGHT

Metrics: nDCG@10 Recall@20 Recall@50 Recall@100 QPS encoding time

Baselines: Static Embedding BM25 USTAD BGE-m3 SPLADE-v3 LLM2Vec E5-Mistral Full symmetric LLM retrievers

Builds on: tDRO SPLADE E5-Mistral LLM2Vec Matryoshka Representation Learning

Closest papers by meaning

• RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.91)
• Demystifying and Enhancing the Efficiency of Large Language Model Based Search Agents (ICLR 2026 · similarity 0.88)
• Search for Efficient Large Language Models (NeurIPS 2024 · similarity 0.87)
• Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval (NeurIPS 2025 · similarity 0.87)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.87)
• Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval (AAAI 2025 · similarity 0.86)