Research map / Large language models / Language, safety and interpretability
Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SparseCBM, which applies concept-specific second-order unstructured pruning to LLM backbones within a concept bottleneck framework, yielding input-, subnetwork-, and concept-level explanations plus a sparsity-based inference-time intervention that edits concept masks.
Problem
Existing interpretability methods for LLMs give either local or global explanations within one dimension; concept bottleneck models for LLMs lack granular, token- and subnetwork-level reasoning and cannot correct backbone errors at inference time.
Approach
Prune the LLM backbone into a separate 0/1 weight-mask subnetwork per concept, using Hessian-based (dampened empirical Fisher) second-order pruning under a joint concept+task loss, and at test time drop/grow low/high-saliency weights in the mispredicted concept's mask while keeping the backbone frozen.
Key results
- On CEBaB with BERT, SparseCBM reaches 83.5/85.6 concept accuracy/F1 versus 83.2/85.3 for dense CBM, and 66.9/79.1 task accuracy/F1 versus 66.9/78.1.
- On IMDB-C with OPT-1.3B, SparseCBM gets task accuracy 86.6 and F1 83.9 versus 85.7/83.3 for the dense CBM.
- GPT-4 prompting performs notably worse (e.g., CEBaB task 51.3 accuracy vs 63.9-73.8 for fine-tuned BERT-family models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace CBM with SparseCBM using the same OPT-1.3B backbone | OPT-1.3B CBM · CEBaB concept prediction | Accuracy / Macro F1: 90.0/91.5 -> 89.9/91.6 | Mixed |
| Replace CBM with SparseCBM using the same OPT-1.3B backbone | OPT-1.3B CBM · CEBaB task prediction | Accuracy / Macro F1: 73.6/82.1 -> 73.8/82.6 | Helped |
| Replace CBM with SparseCBM using the same OPT-1.3B backbone | OPT-1.3B CBM · IMDB-C concept prediction | Accuracy / Macro F1: 76.8/74.6 -> 76.4/74.7 | Mixed |
| Replace CBM with SparseCBM using the same OPT-1.3B backbone | OPT-1.3B CBM · IMDB-C task prediction | Accuracy / Macro F1: 85.7/83.3 -> 86.6/83.9 | Helped |
| Replace CBM with SparseCBM using the same DistilBERT backbone | DistilBERT CBM · CEBaB concept prediction | Accuracy / Macro F1: 81.1/83.5 -> 82.0/84.0 | Helped |
| Replace CBM with SparseCBM using the same DistilBERT backbone | DistilBERT CBM · CEBaB task prediction | Accuracy / Macro F1: 63.9/76.5 -> 64.7/77.1 | Helped |
| Replace CBM with SparseCBM using the same DistilBERT backbone | DistilBERT CBM · IMDB-C concept prediction | Accuracy / Macro F1: 67.5/63.8 -> 68.4/64.3 | Helped |
| Replace CBM with SparseCBM using the same DistilBERT backbone | DistilBERT CBM · IMDB-C task prediction | Accuracy / Macro F1: 76.5/69.8 -> 76.9/71.4 | Mixed |
| Apply sparsity-based inference-time intervention to correct a mispredicted concept by adapting its mask | SparseCBM with DistilBERT · CEBaB | Concept and task prediction accuracy | Helped |
| Use a 1% weight adjustment proportion for sparsity-based intervention | SparseCBM with DistilBERT · CEBaB | Prediction accuracy: 1% | Helped |
| Increase target sparsity beyond its suitable level | SparseCBM · CEBaB | Concept and task prediction performance | Hurt |
Surprising findings
- Directly prompting GPT-4 yields clearly inferior concept and task predictions compared with fine-tuned small models.
- Sparsity-based intervention reached its best setting at only about 1% of weights modified (≈2% of DistilBERT backbone parameters).
Limitations
- Concept-specific pruning increases training time.
- Requires concept annotations, and the framework is sensitive to noisy concept and task labels.
- IMDB-C training set is very small (100 examples), and results are averages of three runs with modest margins over CBM.
Experimental setup
Datasets: CEBaB IMDB-C
Benchmarks: CEBaB IMDB-C
Metrics: accuracy macro F1
Baselines: GPT4 prompting Standard fine-tuning CBM (joint)
Builds on: Concept Bottleneck Models Optimal Brain Compression / second-order pruning joint CBM training
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment (NeurIPS 2024 · similarity 0.90)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.90)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.90)
- LLM Layers Immediately Correct Each Other (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.