Research map / Large language models / Language, safety and interpretability
SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?
Michael Kirchhof, Luca Füger, Adam Golinski, Eeshan Gunesh Dhekane, Arno Blaas, Seong Joon Oh, Sinead Williamson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Wasserstein distance over the whole logit vector instead of only the masked word log-prob (ablation) | SelfReflect · Good vs almost-good summaries | pairwise accuracy: 85.90% -> 93.20% | Improved |
| Add the one-by-one masked-out task instead of comparing full-answer log-likelihoods (ablation) | SelfReflect · Good vs almost-good summaries | pairwise accuracy: 33.64% -> 93.20% | Improved |
| Put sampled answers into the judge context instead of the sampling-free comparison (ablation) | SelfReflect · Good vs almost-good summaries | pairwise accuracy: 54.85% -> 93.20% | Improved |
| Use generative masked-out task instead of discriminative P(True) (ablation) | SelfReflect · Human preference alignment | Krippendorff alpha: -0.032 -> 0.690 | Improved |
| Use generative masked-out task instead of discriminative P(True) (ablation) | SelfReflect · Good vs bad summaries | pairwise accuracy: 65.29% -> 98.77% | Improved |
| Sample & summarize with N=10 instead of Greedy | Qwen2.5 72B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 91 -> 85 | Improved |
| Increase samples from N=10 to N=20 | Sample & summarize, Qwen2.5 72B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 85 -> 84 | No clear effect |
| Use Basic prompt asking for all options instead of Greedy | Qwen2.5 72B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 91 -> 94 | Hurt |
| Use CoT prompt instead of Greedy | Qwen2.5 72B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 91 -> 96 | Hurt |
| Use reasoning before summarizing instead of greedy answer | QwQ 32B · Self-reflection benchmark, RLVR models | SelfReflect score (lower better): 96 -> 105 | Hurt |
| Sample & summarize N=10 instead of Greedy for the smallest model | Qwen2.5 0.5B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 96 -> 96 | No clear effect |
| Use Sample & summarize instead of Greedy | Gemma 3 1B Instruct · Self-reflection benchmark | SelfReflect score (lower better): 116 -> 117 | No clear effect |
Closest papers by meaning
- Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (ICLR 2024 · similarity 0.89)
- To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.89)
- Evaluating language models as risk scores (NeurIPS 2024 · similarity 0.88)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.88)
- Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.88)
- To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
