Research map / Architectures and efficiency / Attention and transformers

Contrastive Retrieval Heads for Improved Attention-Based Reranking

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CoRe heads, a small set of attention heads selected by a contrastive softmax-style score over positive vs. hard-negative documents, for zero-shot list-wise attention-based re-ranking with layer pruning for efficiency.

Problem

Attention-based list-wise re-rankers aggregate attention over all heads, which adds noise and redundancy; prior retrieval-head selection (QR heads) uses absolute attention to the positive document and ignores competing negatives.

Approach

Score each attention head with a contrastive softmax over the positive document versus hard negatives, select the top ~8 heads (under 1%), and aggregate only their attention for re-ranking; the selected heads concentrate in middle layers, enabling pruning of the final 50% of layers.

Key results

• On BEIR top-60, CoRe-R averages 50.8 nDCG@10 with Mistral 7B vs 48.6 for QR-R and 47.8 for ICR.
• With Llama-3.1 8B on BEIR top-40, CoRe-R reaches 52.1 average nDCG@10 vs 51.3 for QR-R and 50.2 for ICR.
• Pruning 50% of layers keeps near-identical re-ranking accuracy while cutting GPU memory by about 40% and latency by about 20%.

Surprising findings

• QR-R drops sharply on Quora (e.g., 71.4 vs. 86.7 retriever baseline for Mistral 7B), while CoRe-R stays high, illustrating the failure of absolute-attention head selection on near-duplicate negatives.
• For Granite-3.2 8B, nearly all heads contribute little noise, so the number of aggregated heads has negligible effect on accuracy, unlike the other models.

Limitations

• Attention-based re-ranking requires access to full attention weights, restricting it to open-weight models.
• Mistral 7B results depend strongly on temperature, and Quora/near-duplicate tasks still need task-specific prompts for best results.
• Several BEIR datasets (e.g., SciDocs, Climate-FEVER) still underperform the retriever baseline in some settings.

Experimental setup

Datasets: Natural Questions (NQ) training set MSMARCO (ablation for head detection) BEIR MLDR MuSiQue CLIPPER

Benchmarks: BEIR MLDR MuSiQue CLIPPER

Metrics: nDCG@10 Recall@2 Recall@5 latency peak GPU memory

Baselines: ICR QR-R NIAH-R retriever baseline (granite-embedding-30m-english)

Builds on: In Context Re-ranking (ICR) query-focused retrieval heads (QR heads) retrieval heads (Needle-in-a-Haystack) InfoNCE

Closest papers by meaning

• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.90)
• AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking (NeurIPS 2025 · similarity 0.89)
• DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.87)
• UniRank: Unified List-wise Reranking via Confidence-Ordered Denoising (NeurIPS 2026 · similarity 0.87)
• Scalable In-context Ranking with Generative Models (NeurIPS 2025 · similarity 0.87)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.87)