Research map / Large language models / Retrieval and knowledge

Retrieval from Within: An Intrinsic Capability of Attention-Based Models

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: RAG robustness and evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes INTRA, which reuses a frozen T5Gemma2 encoder-decoder's decoder cross-attention queries (with learnable retrieval tokens and Reverse-QWK) to score pre-encoded corpus chunks via MaxSim, then reuses those encoder states as generation context.

Problem

Standard RAG separates retrieval from generation with distinct representation spaces and re-encodes retrieved evidence per query; the paper asks whether an attention-based encoder-decoder can retrieve evidence from its own internal representations for QA.

Approach

Decoder cross-attention queries (augmented with trainable retrieval tokens and layer-aggregation weights) are scored by MaxSim against pre-encoded chunk states; Reverse-QWK moves key projections to the query side so one normalized encoder pool serves all layers, letting the same states be used for both retrieval and generation.

Key results

• INTRA reaches 70.1% complete-evidence recall@5 on HotPotQA versus 54.8% for BGE (best dense baseline), per the complete-evidence recall table.
• End-to-end average EM of 42.2 and F1 of 50.9 across four benchmarks, versus 39.2/47.5 for Qwen3-Emb-4B plus reranker; INTRA leads on HotPotQA, 2Wiki, and MuSiQue but trails on NQ (53.7 vs 55.1 EM).
• Using the T5Gemma2-4B decoder as generator gives the largest average gap closure (60.2%) versus 58.8% for Qwen3.5-9B.

Surprising findings

• A frozen decoder pretrained only for generation, with only ~164K trainable retrieval parameters, outperforms retrieval-pretrained dense embedders on multi-hop QA recall.
• Removing the top-ranked INTRA chunk lowers EM by 7.2 points, while chunks ranked 6-10 show effects indistinguishable from zero in the leave-one-out analysis.

Limitations

• Evaluated only on a fixed ~100M-token pool and a single T5Gemma2-style encoder-decoder family; not shown to generalize to general-purpose retrievers or open-web settings.
• Excludes decoder-only models; restricted to text QA with short answers.
• TTFT excludes retrieval time; storage (about 512 GB for 100M tokens at bf16) and index scaling costs may change end-to-end trade-offs.

Experimental setup

Datasets: HotPotQA 2WikiMultihopQA MuSiQue Natural Questions CLaRa QA pretraining data Wikipedia

Benchmarks: HotPotQA 2WikiMultihopQA MuSiQue Natural Questions

Metrics: complete-evidence recall@k exact match F1 gap closure time-to-first-token

Baselines: TF-IDF BM25 BGE-large Qwen3-Embedding-0.6B Qwen3-Embedding-4B Jina reranker Reciprocal rank fusion hybrid RAG ColBERT-style MaxSim

Builds on: T5Gemma2 ColBERT BM25 CLaRa

Closest papers by meaning

• The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation (NeurIPS 2025 · similarity 0.89)
• Retrieval Head Mechanistically Explains Long-Context Factuality (ICLR 2025 · similarity 0.88)
• Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection (ICLR 2025 · similarity 0.87)
• Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads (NeurIPS 2026 · similarity 0.87)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (ICLR 2024 · similarity 0.87)
• RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.87)