Research map / Large language models / Language, safety and interpretability

Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract)

AAAI 2024 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a task-agnostic automated framework that evaluates free-text LLM explanations on fidelity (textual perturbation consistency, forward simulatability utility) and interpretability (LM-based coherence, cognitive fluency, cognitive salience), and applies it to GPT-3.5 on stance detection.

Problem

Rigorously assessing the fidelity and interpretability of free-text rationales from proprietary LLMs, where internal features are inaccessible and human evaluation is costly and subjective.

Approach

Measure fidelity via textual input perturbations (no model internals needed) and interpretability via language models acting as evaluators in place of human annotators, decomposed into coherence, cognitive fluency, and cognitive salience.

Key results

• GPT-3.5 with 4-shot CoT reaches faithfulness 0.67 vs 0.61 at 0-shot, with utility 0.24-0.26 and cognitive fluency about 0.89-0.91 across settings.
• F-CoT (prompting for faithful explanations) lowers faithfulness from 0.44 (0-shot) to 0.50 (4-shot) relative to CoT, while improving rationale coherence to 0.80-0.81.
• Rationale coherence drops from 0.79 (CoT 0-shot) to 0.62 (CoT 4-shot).

Measured effects reported in the paper

Surprising findings

• Prompting GPT-3.5 explicitly for faithful explanations reduced measured faithfulness, which the authors attribute to a misunderstanding of faithfulness.
• Adding few-shot examples to CoT raised faithfulness but lowered rationale coherence.

Limitations

• Evaluated only one model (GPT-3.5) on one task (stance detection) in a short student abstract.
• Specific evaluator models and fidelity formulas are not detailed in the text.

Experimental setup

Metrics: faithfulness utility rationale coherence cognitive fluency cognitive salience

Baselines: chain-of-thought prompting

Builds on: forward simulatability chain-of-thought

Closest papers by meaning

• Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.89)
• Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.89)
• Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.88)
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models (ICML 2024 · similarity 0.88)
• Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals (ICLR 2024 · similarity 0.88)
• Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.87)