Research map / Large language models / Reasoning and chain of thought
AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AcuRank, which uses TrueSkill Bayesian relevance estimates to adaptively select uncertain documents for listwise LLM reranking, improving the accuracy-cost trade-off over fixed sliding-window and tournament baselines.
Problem
Fixed-computation listwise LLM reranking (sliding windows, tournaments) ignores query difficulty and document distribution, wasting calls on easy queries and under-allocating to ambiguous ones.
Approach
Model each document's relevance as a Gaussian belief with TrueSkill, compute top-k membership probabilities via an expectation threshold, and iteratively rerank only the documents with intermediate top-k probability, updating beliefs from listwise reranker outputs.
Key results
- AcuRank averages 55.5 NDCG@10 across TREC-DL and BEIR with 19.7 reranker calls, versus 54.6 for SW-3 at 26.4 calls (RankZephyr, BM25 top-100).
- AcuRank-9 matches SW-1's average call count (8.8) while achieving 54.6 vs 54.3 NDCG@10.
- Number of reranker calls correlates negatively with WIG query difficulty (rho = -0.27, p < 1e-8), so harder queries receive more compute.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| allocate reranking adaptively using uncertainty estimates | AcuRank versus fixed-computation rerankers · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (Avg.): 54.6 -> 55.5 | Helped |
| use adaptive computation on BM25 top-1000 candidates | AcuRank versus fixed-computation baselines · TREC-DL and BEIR, BM25 top-1000, RankZephyr-7B | NDCG@10 (Avg.): 58.0 vs 56.2 | Helped |
| use adaptive computation with SPLADE++ED retrieval scores | AcuRank · BEIR, SPLADE++ED top-100, RankZephyr-7B | NDCG@10 (Avg.): 52.6 -> 53.8 | Helped |
| use adaptive computation with Contriever retrieval scores | AcuRank · BEIR, Contriever top-100, RankZephyr-7B | NDCG@10 (Avg.): 46.0 -> 47.4 | Helped |
| use adaptive computation with RankGPT (gpt-4.1-mini) | AcuRank · BEIR, BM25 top-100 | NDCG@10 (Avg.): 53.4 -> 53.7 | Helped |
| initialize TrueSkill scores from first-stage retrieval scores (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (All): 54.8 -> 55.5 | Helped |
| partition uncertain documents sequentially instead of randomly (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (All): 55.3 -> 55.5 | Helped |
| partition uncertain documents sequentially instead of randomly (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | reranker calls: 22.6 -> 19.7 | Helped |
| stop when uncertain documents fall below threshold instead of using top-k stability (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (All): 55.2 -> 55.5 | No clear effect |
| stop when uncertain documents fall below threshold instead of using top-k stability (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | reranker calls: 22.7 -> 19.7 | Helped |
| use σ=μ/3 instead of fixed uncertainty σ=8 (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (All): 55.1 -> 55.5 | Helped |
| use σ=μ/3 instead of σ=μ/2 (ablation) | AcuRank · TREC-DL and BEIR, BM25 top-100, RankZephyr-7B | NDCG@10 (All): 55.4 -> 55.5 | Helped |
Surprising findings
- Fixed-threshold ablation without first-stage initialization used fewer calls (13.4) but lower accuracy, showing initialization trades compute for accuracy.
- Reported call counts disadvantage AcuRank relative to baselines in latency and FLOPs terms because its windows are smaller, so call count alone understates efficiency gains.
Limitations
- Efficiency reported mainly as reranker call counts, with latency and FLOPs on a subset only.
- Approximate rank-probability computation relies on independence and large-n assumptions.
- Hyperparameters tuned on a subset of DL19 and DL20 before applying to other datasets.
Experimental setup
Datasets: TREC-DL 2019 TREC-DL 2020 TREC-DL 2021 TREC-DL 2022 TREC-DL 2023 DL-Hard BEIR (TREC-COVID, NFCorpus, Signal-1M, TREC-News, Robust04, Touche, DBPedia, SciFact)
Benchmarks: TREC-DL 2019 TREC-DL 2020 TREC-DL 2021 TREC-DL 2022 TREC-DL 2023 DL-Hard BEIR
Metrics: NDCG@10 number of reranker calls latency input length petaFLOPs
Baselines: Sliding windows (SW-1, SW-2, SW-3) TourRank-1 TourRank-2 TourRank-5 TourRank-10 TrueSkill-Static RankVicuna-7B
Builds on: TrueSkill RankGPT sliding window reranking TourRank RankLLM
Closest papers by meaning
- Contrastive Retrieval Heads for Improved Attention-Based Reranking (NeurIPS 2026 · similarity 0.89)
- DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation (NeurIPS 2025 · similarity 0.88)
- Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers (ICLR 2025 · similarity 0.87)
- UniRank: Unified List-wise Reranking via Confidence-Ordered Denoising (NeurIPS 2026 · similarity 0.87)
- Very Efficient Listwise Multimodal Reranking for Long Documents (ICML 2026 · similarity 0.87)
- GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.