Research map / Large language models / Reasoning and chain of thought
The Overthinking Predicament: When Reasoning Hurts Ranking
Xuan Lu, Haohang Huang, Rui Meng, Yaohui Jin, Wenjun Zeng, Xiaoyu Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper examines whether explicit chain-of-thought reasoning helps LLM-based document rerankers, finding that it can hurt ranking quality (the "overthinking" predicament).
Problem
Document reranking in information retrieval; recent work adds explicit chain-of-thought reasoning to LLM rerankers, but its effectiveness for ranking is underexplored.
Approach
Systematic study of explicit CoT reasoning in LLM-based rerankers, showing that reasoning can degrade ranking quality rather than improve it. Abstract is truncated, so details of the proposed method are not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add chain-of-thought reasoning to pointwise reranker (4B) (ablation) | Direct-Point-4B · BRIGHT (original query) | NDCG@10: 25.5 -> 16.5 | Hurt |
| add chain-of-thought reasoning to pointwise reranker (8B) (ablation) | Direct-Point-8B · BRIGHT (original query) | NDCG@10: 26.8 -> 20.7 | Hurt |
| add chain-of-thought reasoning to pointwise reranker (ablation) | Direct-Point · BEIR | NDCG@10: 45.4 -> 40.1 (4B); 43.7 -> 39.3 (8B) | Hurt |
| add reasoning to listwise reranker (SFT+GRPO, 8B) (ablation) | Direct-List-8B · BRIGHT (original query) | NDCG@10: 27.1 -> 25.9 | Hurt |
| add reasoning to listwise reranker (SFT+GRPO, 8B) (ablation) | Direct-List-8B · BEIR | NDCG@10: 41.8 -> 39.9 | Hurt |
| add reasoning to pointwise reranker (ablation) | Direct-Point-8B · BEIR calibration | ECE: 0.105 -> 0.151 | Hurt |
| add reasoning to pointwise reranker (ablation) | Direct-Point-8B · BRIGHT-Biology class-conditional | TPR/TNR: TPR 31.1 -> 50.5; TNR 100.0 -> 98.1 | Mixed |
| add GRPO after SFT | listwise rerankers · BRIGHT (original query) | NDCG@10: 24.9 -> 27.1 (Direct-List-8B) | Improved |
| add reasoning to listwise reranker (ablation) | Direct-List-8B (SFT) · training split (100 instances) | NDCG@10: 80.41 -> 82.57 mean, std 2.1 -> 3.2 | Mixed |
| add reasoning to listwise reranker (4B GRPO) (ablation) | Direct-List-4B · MS MARCO DL19 | NDCG@10: 73.77 -> 70.76 | Hurt |
| GRPO instead of SFT for Reason-List rationale length (ablation) | Reason-List-8B · training split | rationale tokens: 397.7 -> 172.3 | Improved |
Surprising findings
- Explicit chain-of-thought reasoning can hurt LLM-based document ranking quality, the 'overthinking' effect.
Closest papers by meaning
- AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking (NeurIPS 2025 · similarity 0.90)
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.88)
- TFRank: Think-Free Reasoning Enables Practical Pointwise LLM Ranking (AAAI 2026 · similarity 0.87)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.86)
- Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval (ICLR 2026 · similarity 0.86)
- UniRank: Unified List-wise Reranking via Confidence-Ordered Denoising (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
