Research map / Large language models / Language, safety and interpretability
Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLM-generated counterfactuals for model-agnostic causal explanations of black-box NLP models, plus a causal embedding matching method trained with LLM-generated CFs that is far cheaper at inference, and a new LLM-built stance detection explanation benchmark.
Problem
Faithful, model-agnostic causal explanations of NLP model predictions for high-level concepts are hard to obtain: counterfactuals are unavailable, LLM-based CF generation is costly at inference, and existing benchmarks are scarce.
Approach
Prompt an LLM to generate counterfactuals that change one concept while holding confounders fixed, and use these (plus matched and misspecified examples) to train a causal sentence encoder whose embedding space supports fast nearest-neighbor matching as CF approximation; also proves approximate-CF methods are order-faithful while non-causal ones need not be.
Key results
- Zero/few-shot and fine-tuned generative CF models yield lowest explanation error on CEBaB (fine-tuned T5 avg L2 .28 at K=1 vs causal matching .52), making generative models SOTA model-agnostic explainers.
- The causal matching model beats all matching baselines at K=1 (average L2 .52 vs .58 for Approx) and K=10 (.46 vs .49 for Approx).
- Top-K matching lowers error for every method; matching latency is about 1000x faster than generative baselines for batches of 100 examples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Top-K matching (K=10) instead of a single match (K=1) | CEBaB explainers · CEBaB | L2 error: Causal Model .66 -> .56 (DistilBERT); Fine-tune Generative .38 -> .32 (DistilBERT) | Helped |
| Use few-shot rather than zero-shot prompts | Generative CF explanation · CEBaB | L2 error (AVG): .32 -> .30 | Helped |
| Fine-tune T5 on paired human-written CFs rather than use few-shot ChatGPT | Generative CF explanation · CEBaB | L2 error (AVG): .30 -> .28 | Helped |
| Use the causal representation model rather than the strongest matching baseline | CEBaB matching · CEBaB | L2 error (AVG): .58 -> .52 | Helped |
| Replace the standard candidate set with one containing ground-truth CFs | Causal Model · CEBaB | L2 error (AVG): .52 -> .09 | Helped |
| Replace the standard candidate set with one containing ground-truth CFs | Causal Model · Stance Detection | L2 error (AVG): 0.59 -> 0.10 | Helped |
| Replace SentenceTransformer with RoBERTa as the backbone (ablation) | Causal Model · CEBaB, original candidate set | L2 error: .65 -> .66 | No clear effect |
| Remove filtering of unsuccessful or misspecified CFs (ablation) | Causal Model · CEBaB, candidate set augmented with misspecified CFs | L2 error: .70 -> .77 | Hurt |
| Replace human-annotated concepts with zero-shot LLM concept predictions (ablation) | Causal Model · CEBaB, original candidate set | L2 error: .65 -> .65 | No clear effect |
| Remove XCF examples from the objective (ablation) | Causal Model · CEBaB, candidate set with ground-truth CFs | L2 error: .13 -> .26 | Hurt |
| Remove XM examples from the objective (ablation) | Causal Model · CEBaB, candidate set with misspecified CFs | L2 error: .70 -> .81 | Hurt |
| Remove XMiCF examples from the objective (ablation) | Causal Model · CEBaB, candidate set with misspecified CFs | L2 error: .70 -> .83 | Hurt |
Surprising findings
- Causal matching with ground-truth CFs in the candidate set outperforms even the generative models.
- Replacing human-annotated concepts with zero-shot LLM-predicted concepts leaves matching performance essentially unchanged.
Limitations
- Requires an identifiable, annotated causal graph and annotated training/candidate sets for matching.
- Matching quality depends heavily on the candidate set; without ground-truth-like CFs in candidates it lags generative methods.
- Stance benchmark labels and counterfactuals come from GPT-4, and LLMs may mismodel mediator effects.
Experimental setup
Datasets: CEBaB SemEval 2016 stance detection CEBaB inclusive train set
Benchmarks: CEBaB SemEval 2016 stance detection (new explanation setup)
Metrics: L2 error cosine distance error norm difference error latency
Baselines: Zero-shot Generative (ChatGPT) Few-shot Generative (ChatGPT) Fine-tune Generative (T5-base) Random Match Propensity Approx PT RoBERTa PT S-Transformer FT S-Transformer
Builds on: CEBaB Causal Proxy Model all-mpnet-base-v2 SentenceTransformer RoBERTa
Closest papers by meaning
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICML 2024 · similarity 0.89)
- Walk the Talk? Measuring the Faithfulness of Large Language Model Explanations (ICLR 2025 · similarity 0.89)
- Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention (AAAI 2024 · similarity 0.88)
- A General Search-Based Framework for Generating Textual Counterfactual Explanations (AAAI 2024 · similarity 0.88)
- Automated Assessment of Fidelity and Interpretability: An Evaluation Framework for Large Language Models’ Explanations (Student Abstract) (AAAI 2024 · similarity 0.88)
- Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.