Research map / Large language models / Language, safety and interpretability

Explanations that reveal all through the definition of encoding

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines encoding in feature-attribution explanations as conditional dependence between the selection indicator and the label given explanation values, shows ROAR and FRESH fail to weakly detect it and EVAL-X is only weak, and proposes STRIPE-X, a strong detector that is shown to flag encoding in LLM-generated sentiment explanations.

Problem

Evaluations of feature attributions score explanations by how well the explanation predicts the label, but explanations can encode label information in their selection pattern, so such evaluations can reward explanations whose selected values are not actually predictive; no general characterization

Approach

Define encoding as a conditional dependence between the label and the explanation-selection indicator given the selected values; show non-encoding explanations are "what you see is what you get", then combine EVAL-X with a conditional mutual information penalty (STRIPE-X) so that encoding explanations are pushed below all non-encoding ones.

Key results

• On the image task, ROAR and FRESH score the predictive-encoding and marginal-encoding explanations as high as the optimal explanation (0.69 and -0.23 respectively), so they are not weak detectors.
• EVAL-X scores the marginal-encoding explanation at -0.53, above the fixed non-encoding explanation at -0.64, showing it is not a strong detector, while STRIPE-X places all encoding explanations below the -0.69 negative-entropy threshold.
• STRIPE-X score for Llama 3 sentiment explanations is -2.78, below the negative marginal entropy of -0.69, indicating the LLM explanations encode.

Measured effects reported in the paper

Surprising findings

• Despite being prompted to select all words it based its selection on, Llama 3 explanations still encode, mostly by omitting the control-flow sentence.
• ROAR and FRESH assign optimal scores to encoding explanations in both simulated and image settings.

Limitations

• STRIPE-X depends on misestimated models (e.g., EVAL-X surrogate), which can cause errors; alpha is set heuristically to 20.
• Analysis restricted to feature-subset explanations; free-text rationales are not covered.

Experimental setup

Datasets: cats_vs_dogs (TensorFlow) simulated discrete and hybrid DGPs sentiment movie review templates

Benchmarks: simulated DGPs cats_vs_dogs image construction LLM sentiment explanations

Metrics: ROAR log-likelihood FRESH log-likelihood EVAL-X log-likelihood STRIPE-X score encode-meter

Baselines: ROAR FRESH EVAL-X REAL-X

Builds on: ROAR FRESH EVAL-X REAL-X conditional independence testing

Closest papers by meaning

• Dissenting Explanations: Leveraging Disagreement to Reduce Model Overreliance (AAAI 2024 · similarity 0.83)
• Selective Explanations (NeurIPS 2024 · similarity 0.83)
• Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.83)
• How to Probe: Simple Yet Effective Techniques for Improving Post-hoc Explanations (ICLR 2025 · similarity 0.83)
• Efficient and Accurate Explanation Estimation with Distribution Compression (ICLR 2025 · similarity 0.83)
• Training Language Models to Explain Their Own Computations (NeurIPS 2026 · similarity 0.82)