Research map / Large language models / Language, safety and interpretability
Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search
Ekaterina Fadeeva, Maiya Goloburda, Aleksandr Rubashevskii, Roman Vashurin, Artem Shelmanov, Preslav Nakov, Mrinmaya Sachan, Maxim Panov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace multinomial sampling with probability-weighted beam search candidates | Dissimilarity · 6 QA datasets avg, Gemma 3 4B instruct | PRR: .206 -> .252 | Improved |
| replace multinomial sampling with beam search candidates | Eccentricity · 6 QA datasets avg, Qwen 3 8B instruct | PRR: .251 -> .345 | Improved |
| replace multinomial sampling with beam search candidates | CocoaPPL · 6 QA datasets avg, Qwen 3 8B base | PRR: .461 -> .461 | No clear effect |
| replace multinomial sampling with beam search candidates | Dissimilarity · Qwen 3 8B base, 6 QA datasets avg | PRR: .477 -> .478 | No clear effect |
| increase number of candidates M under beam search (ablation) | Dissimilarity, Eccentricity, EigVecDissimilarity · TriviaQA, Gemma 3 4B base | PRR | Improved |
| use beam search with M=1 (greedy) (ablation) | Dissimilarity · TriviaQA, Gemma 3 4B base | PRR | Hurt |
| use beam search instead of multinomial sampling on longer generations (ablation) | Dissimilarity · TriviaQA and CoQA, Gemma 3 4B base | PRR | No clear effect |
| use diverse beam search instead of plain beam search (ablation) | Dissimilarity, Eccentricity, EigVecDissimilarity · TriviaQA and CoQA, Gemma 3 4B base and Llama 3.1 8B base | PRR | Hurt |
| mix beam candidates with multinomial samples (hybrid) (ablation) | Dissimilarity · TriviaQA and CoQA, Gemma 3 4B base and Llama 3.1 8B base | PRR: .771 -> .784 | Mixed |
| add a probability floor epsilon to beam weights (toward uniform weights) (ablation) | beam-weighted estimators · TriviaQA and CoQA, Gemma 3 4B base and Llama 3.1 8B base | PRR | No clear effect |
| apply beam candidates to Semantic Entropy (ablation) | Semantic Entropy · 6 QA datasets avg, Gemma 3 4B base | PR-AUC: .401 -> .472 | Improved |
| apply beam candidates to Degree Matrix and SAR (ablation) | Degree Matrix, SAR · 6 QA datasets avg, Gemma 3 4B base | PR-AUC: .469 -> .467 | No clear effect |
Closest papers by meaning
- Conformal Prediction with Paraphrase-Aware Scoring for LLM Uncertainty Quantification (NeurIPS 2026 · similarity 0.89)
- Benchmarking LLMs via Uncertainty Quantification (NeurIPS 2024 · similarity 0.88)
- SPANUQ: Span-Level Uncertainty Quantification for Large Language Model Generation (NeurIPS 2026 · similarity 0.88)
- FLUE: Streamlined Uncertainty Estimation for Large Language Models (AAAI 2025 · similarity 0.88)
- Calibrating Large Language Models with Sample Consistency (AAAI 2025 · similarity 0.87)
- An Empirical Analysis of Uncertainty in Large Language Model Evaluations (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
