Research map / Large language models / Language, safety and interpretability

Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Statistical frameworks for LLM evaluation · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes causal concept faithfulness, measuring alignment between LLM explanation-implied concept effects and counterfactual causal concept effects, estimated with LLM-generated counterfactuals and a Bayesian hierarchical model, applied to GPT-3.5, GPT-4o, and Claude-3.5-Sonnet on BBQ and MedQA.

Problem

LLM explanations can be plausible yet unfaithful, misrepresenting which input factors actually influenced the model's answer; existing faithfulness measures give scores but not the semantic patterns of unfaithfulness.

Approach

Define faithfulness as the Pearson correlation between concept-level causal effects (KL divergence of answer distributions under concept counterfactuals) and the rate at which explanations cite each concept; estimate causal effects using an auxiliary LLM to create realistic counterfactuals and a Bayesian hierarchical model that partially pools across questions and concept categories.

Key results

• On BBQ, GPT-3.5 has the highest dataset-level faithfulness (0.75, 90% CI [0.42, 1.00]), above Claude-3.5-Sonnet (0.62) and GPT-4o (0.56).
• On MedQA, GPT-3.5 scores 0.50 (CI [0.18, 0.77]) versus 0.34 for GPT-4o and 0.30 for Claude-3.5-Sonnet.
• The method uncovers unfaithful explanations that hide safety-alignment-driven refusals and social bias, e.g., GPT models shift toward the 'rich' individual while never citing income.

Measured effects reported in the paper

Surprising findings

• GPT-3.5 is the most faithful of the models on BBQ, despite being the older and weaker model.
• GPT models' explanations never mention the relative incomes of individuals even when income swaps change answers toward the rich individual.

Limitations

• Dataset-level results use only 30 questions per dataset, so estimates may not be representative (wide credible intervals).
• Relies on an auxiliary LLM (GPT-4o) for concept extraction, counterfactuals, and explanation analysis, which sometimes makes errors and needs dataset-specific prompts.
• Fails to handle correlated concepts; evaluated only on closed-source models apart from a brief Llama-3.1-8B check.

Experimental setup

Datasets: BBQ (30 sampled questions) MedQA (30 sampled Type 2 questions)

Benchmarks: BBQ MedQA

Metrics: Pearson correlation (causal concept faithfulness) KL divergence (causal concept effect) credible intervals

Baselines: GPT-4o GPT-3.5 Claude-3.5-Sonnet

Builds on: BBQ MedQA chain-of-thought prompting

Closest papers by meaning

• To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts (ICLR 2025 · similarity 0.90)
• Aligning Large Language Models for Faithful Integrity Against Opposing Argument (AAAI 2025 · similarity 0.90)
• Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.89)
• Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICML 2024 · similarity 0.89)
• Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.89)
• Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.89)