Research map / Large language models / Language, safety and interpretability
What LLMs Explain Is Not What They Believe: Evaluating Explanation Sufficiency Under Models' Own Input Beliefs
Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add few-shot examples with contradiction to the alteration prompt (ablation) | SCSuff · IMDB (Qwen and Llama) | SCSuff: drops near zero | Hurt |
| add natural few-shot IMDB examples to the alteration prompt (ablation) | SCSuff · IMDB | SCSuff | No clear effect |
| add few-shot examples with contradiction to the alteration prompt (ablation) | SCSuff · IMDB (Ministral) | SCSuff: decreases substantially, not to zero | Hurt |
| increase number of alternative inputs (ablation) | SCSuff · MMLU + authority and IMDB, 8B models | SCSuff: max diff 0.038 | No clear effect |
| paraphrase the alteration prompt (ablation) | SCSuff · MMLU + authority and IMDB, 8B models | SCSuff: max diff 0.057 | No clear effect |
| remove chain-of-thought from predictive model (ablation) | SCSuff · MMLU + authority and IMDB | SCSuff: max diff 0.074 | No clear effect |
| use a different LLM to generate alternative inputs (ablation) | SCSuff · MMLU + authority and IMDB | SCSuff: max diff 0.175 | Hurt |
| use final-token hidden state instead of average hidden state to predict SCSuff (ablation) | probe · MMLU + authority and IMDB, 8B models | probe accuracy: accuracy < 0.8 -> above 0.84 | Improved |
| increase model size | SCSuff · 9 LLMs, 4 datasets | Spearman correlation with SCSuff: rho = -0.23 | No clear effect |
| improve task accuracy | SCSuff · 9 LLMs, 4 datasets | Spearman correlation with SCSuff: rho = 0.07 | No clear effect |
| higher output entropy | SCSuff · 9 LLMs, 4 datasets | Spearman correlation with SCSuff: rho = 0.24 | No clear effect |
Closest papers by meaning
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.90)
- Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.90)
- Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.90)
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICML 2024 · similarity 0.90)
- Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.89)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
