Research map / Large language models / Language, safety and interpretability
Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes in-context re-ranking (ICR), which scores documents by the attention an LLM's query tokens pay to them, calibrated with a content-free "N/A" query, to rerank in two forward passes without generation.
Problem
LLM-based zero-shot re-rankers rely on autoregressive generation, which is costly (O(N) forward passes) and unreliable for open-weight models that struggle with instruction following and output formatting.
Approach
Use attention weights from query tokens to document tokens inside the LLM prompt as relevance signals, aggregated over all query tokens, layers and heads, and calibrated by subtracting scores from a content-free "N/A" query; this needs only two forward passes and always yields a well-formed ranking.
Key results
- ICR (Llama-3.1 8B) outperforms RankGPT (Llama-3.1 8B) by 8.4 micro-average nDCG@10 points on single-hop BEIR subsets (60.4 vs 52.0) and matches RankGPT (GPT-4o mini) micro-average (60.4 vs 60.5).
- ICR (Mistral 7B) outperforms RankGPT (GPT-3.5 Turbo) by 3.3 micro-average points on single-hop tasks (57.3 vs 54.0).
- ICR cuts re-ranking latency by more than 60% compared with RankGPT, and is over two times faster on SciFact with K=100 documents on one RTX A6000 Ada.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use attention-based ICR instead of RankGPT | Mistral 7B re-ranking · BEIR micro-average | nDCG@10: 13.3 points | Helped |
| Use attention-based ICR instead of RankGPT | Llama-3.1 8B re-ranking · Multi-hop average | Recall@2: 5.4%-6.3% | Helped |
| Use attention-based ICR instead of RankGPT | Llama-3.1 8B re-ranking · DBPedia-Entity | nDCG@10: 41.4 -> 35.3 | Hurt |
| Add content-free-query calibration (ablation) | ICR with Llama-3.1 8B · Single-hop macro-average | nDCG@10: 9.3 points | Helped |
| Add content-free-query calibration (ablation) | ICR with Llama-3.1 8B · Multi-hop macro-average | Recall@5: 6.1 percent | Helped |
| Aggregate attention over all query tokens rather than only the last query token (ablation) | ICR with Llama-3.1 8B · Single-hop macro-average | nDCG@10: 13.7 points | Helped |
| Aggregate attention over all query tokens rather than only the last query token (ablation) | ICR with Llama-3.1 8B · Multi-hop macro-average | Recall@5: 17.1 percent | Helped |
Surprising findings
- Llama-3.1 8B with ICR underperforms Mistral 7B with ICR and RankGPT on TREC DL19 and DL20, which the authors suspect is due to fine-tuning on MS MARCO-related re-ranking data skewing attention.
- ICR is vulnerable to lexical bias, underperforming RankGPT on DBPedia-Entity where distractors have high lexical overlap with the query.
Limitations
- ICR requires access to attention weights, so it cannot be applied to API-only proprietary models; RankGPT with proprietary LLMs still performs best.
- ICR suffers from lexical bias, underperforming on DBPedia-Entity and 2WikiMultiHopQA where distractors share query entities.
- Only decoder-only language models were tested.
Experimental setup
Datasets: TREC DL19 TREC DL20 BEIR MuSiQue 2WikiMultiHopQA HotpotQA
Benchmarks: TREC DL19 TREC DL20 BEIR (TREC-COVID, NFCorpus, DBPedia-Entity, SciFact, SciDocs, FiQA, FEVER, Climate-FEVER, NQ) MuSiQue 2WikiMultiHopQA HotpotQA
Metrics: nDCG@10 recall@2 recall@5 All-Recall@5 success rate latency
Baselines: RankGPT ColBERT v2 BM25 RankVicuna Setwise FIRST
Builds on: RankGPT attention sorting FIRST
Closest papers by meaning
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.90)
- Scalable In-context Ranking with Generative Models (NeurIPS 2025 · similarity 0.90)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.90)
- Learning to Rank in Generative Retrieval (AAAI 2024 · similarity 0.89)
- RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs (NeurIPS 2024 · similarity 0.88)
- Language Ranker: A Lightweight Ranking framework for LLM Decoding (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.