Research map / Large language models / Language, safety and interpretability

Scalable In-context Ranking with Generative Models

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BlockRank, which enforces block-sparse attention over instruction, document and query segments plus an auxiliary contrastive attention loss, enabling linear-in-N in-context ranking with attention-based inference on fine-tuned Mistral-7B.

Problem

In-context ranking with LLMs becomes computationally expensive as the number of candidate documents grows, since full self-attention over the concatenated candidate list scales quadratically, and autoregressive decoding of rankings adds latency.

Approach

Restrict document tokens to attend only to their own document and the shared instruction (block-sparse attention), use a permutation-invariant position scheme for documents, and add an InfoNCE loss on attention mass from signal-carrying query tokens to relevant documents at a middle layer, so relevance can be read from prefill attention without decoding.

Key results

• On BEIR (11 datasets, top-100 from Contriever), BlockRank Mistral averages 54.8 nDCG@10, above FIRST (54.3) and RankZephyr (53.7).
• On MSMarco (N=50 in ablation) BlockRank with attention-based inference reaches P@1 29.1 and MRR@10 42.0 vs Full-FT Mistral 28.7 / 38.3.
• At N=100 BlockRank is about 4.7x faster than Full-FT, with latency scaling linearly to 1.15s at N=500.

Measured effects reported in the paper

Surprising findings

• Without the NTP loss, BlockRank's decoding P@1 collapses to 15.8 while its attention-based inference remains at 28.6.
• Full-FT's beam-decoded document ID digits show lower entropy than random IDs, indicating poor diversity in multi-prediction outputs.

Limitations

• Findings demonstrated primarily on a single model architecture (Mistral-7B).
• Robustness of learned attention signals for direct inference across diverse tasks is not established.
• Out-of-domain transfer of models trained on one dataset is considerably lower than in-domain performance (e.g., NQ-trained model gets 18.2 P@1 on MSMarco).

Experimental setup

Datasets: MSMarco passage ranking Natural Questions (NQ320K) BEIR

Benchmarks: BEIR MSMarco NQ

Metrics: nDCG@10 Precision@1 MRR@10 inference latency

Baselines: Full-FT Mistral FIRST RankZephyr RankVicuna Cross-Encoder monoT5-XL ColBERTv2 GTR-XXL Mistral-7B-Instruct Gemini-2.0-flash BM25

Builds on: FIRST RankZephyr RankVicuna Longformer BigBird

Closest papers by meaning

• Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale (NeurIPS 2026 · similarity 0.90)
• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.90)
• Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.87)
• Contextual Document Embeddings (ICLR 2025 · similarity 0.86)
• SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking (NeurIPS 2026 · similarity 0.86)
• Retrieval from Within: An Intrinsic Capability of Attention-Based Models (NeurIPS 2026 · similarity 0.86)