Research map / Large language models / Language, safety and interpretability
Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes counterfactual simulatability metrics (precision and generality) for natural language explanations and finds that GPT-3.5/GPT-4 explanations on StrategyQA and SHP are imprecise and uncorrelated with plausibility.
Problem
Existing simulatability metrics evaluate explanations only on the explained input, where they may leak the answer and do not measure whether explanations help humans predict model behavior on related inputs. The paper asks whether LLM natural language explanations let humans build correct mental mod
Approach
Evaluate an explanation by how well humans, reading only the explanation, can infer the model's outputs on LLM-generated counterfactual inputs, framed as logical entailment. Two metrics: simulation precision (fraction of simulatable counterfactuals where the inferred output matches the model's actual output) and simulation generality (one minus average pairwise similarity of simulatable counterfac
Key results
- GPT-4 CoT explanations achieved 81.1 simulation precision on StrategyQA and 93.0 on SHP, with Post-Hoc at 83.9 and 91.5 respectively.
- Simulation precision showed only a very weak correlation with plausibility (+0.012 Pearson, +0.021 Spearman), versus an inter-annotator plausibility correlation of about +0.388.
- CoT did not clearly outperform Post-Hoc in precision (CoT better by 1.2 points on StrategyQA, worse by 1.3 on SHP), and simulation generality showed no correlation with precision.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4 instead of GPT-3.5 for explanations | CoT and Post-Hoc explanations · StrategyQA | simulation precision: +5.5 precision points | Helped |
| Use GPT-4 instead of GPT-3.5 for explanations | CoT and Post-Hoc explanations · SHP | simulation precision: +6.5 precision points | Helped |
| Use CoT instead of Post-Hoc explanations | GPT-3.5 and GPT-4 explanations · StrategyQA and SHP | simulation precision: 1.2 points; 1.3 points | Mixed |
| Force Post-Hoc explanations to justify the unselected answer | Normal Post-Hoc explanation system · StrategyQA, correct-answer subset | simulation precision: 83.4 -> 38.2; 45.2 precision points | Hurt |
Surprising findings
- Chain-of-thought explanations are not clearly more precise than post-hoc explanations, contrary to the expectation that reasoning-before-answer gives more faithful explanations.
- Simulation precision does not correlate with plausibility, suggesting RLHF-style optimization for human approval may not fix low precision.
Limitations
- Evaluation depends on the distribution of LLM-generated counterfactuals, which likely do not cover all simulatable counterfactuals.
- Specific downstream use cases such as model debugging and AI teaching were not directly evaluated.
- Only English was studied and only classification tasks were evaluated.
Experimental setup
Datasets: StrategyQA Stanford Human Preference (SHP)
Benchmarks: StrategyQA Stanford Human Preference (SHP)
Metrics: simulation precision simulation generality Cohen's kappa inter-annotator agreement BLEU cosine similarity Jaccard similarity Spearman correlation with plausibility
Baselines: Polyjuice Post-Hoc explanations Forced explanations
Builds on: chain-of-thought prompting Polyjuice simulatability
Closest papers by meaning
- Towards Unifying Evaluation of Counterfactual Explanations: Leveraging Large Language Models for Human-Centric Assessments (AAAI 2025 · similarity 0.91)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.89)
- Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.89)
- Counterfactual Explanations for Misclassified Images: How Human and Machine Explanations Differ (Abstract Reprint) (AAAI 2024 · similarity 0.87)
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.87)
- Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.