Research map / Architectures and efficiency / RNNs and normalization
Linear Explanations for Individual Neurons
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that top-activation-only neuron explanations miss most of a neuron's causal impact, and proposes Linear Explanations (sparse linear combinations of concepts fit to the full activation range) plus SigLIP-based simulation scoring for vision neurons.
Problem
Existing automated neuron explanation methods in vision describe only the highest-activating inputs, which account for a small share of a neuron's causal effect; evaluation based on those top activations is therefore also insufficient.
Approach
Describe each neuron as a sparse linear combination of concepts, fit to predict activations over the full activation range using concept-presence probabilities (from labels or SigLIP pseudo-labels), with greedy search to choose a short explanation; evaluate explanations by simulating activations on unseen images with a separate SigLIP simulator, scored by correlation or ablation.
Key results
- Top 0.2% of inputs (beta=0.002) account for only 1.52% of total impact of ResNet-50 neurons, while MILAN-style top-15 inputs explain 0.258%.
- LE(SigLIP) reaches average correlation 0.3772 on ResNet-50 layer4 neurons vs 0.1242 for Network Dissection and 0.1871 for CLIP-Dissect.
- On ResNet-50 layer4 ablation scoring, LE(SigLIP) reaches 0.0727 vs at most 0.0215 for baselines (over 3x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · ResNet-50 (ImageNet), layer4 | correlation: 0.1871 $\pm$ 0.002 -> 0.3772 $\pm$ 0.002 | Helped |
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · ResNet-50 (ImageNet), layer4 | ablation score: 0.0215 $\pm$ 0.0004 -> 0.0727 $\pm$ 0.0008 | Helped |
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · ResNet-18 (Places365) | correlation: 0.2208 $\pm$ 0.005 -> 0.4372 $\pm$ 0.003 | Helped |
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · VGG-16 (CIFAR-100) | correlation: 0.2298 $\pm$ 0.004 -> 0.4970 $\pm$ 0.004 | Helped |
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · ViT-B/16 (ImageNet) | correlation: 0.1722 $\pm$ 0.004 -> 0.3489 $\pm$ 0.005 | Helped |
| use LE (SigLIP) instead of CLIP-Dissect | neuron explanations · ViT-L/32 (ImageNet) | correlation: 0.0549 $\pm$ 0.002 -> 0.2182 $\pm$ 0.004 | Helped |
| replace the original concept set with the 20k-word concept set (ablation) | LE (SigLIP) · ResNet-50 (ImageNet), layer4 | correlation: 0.3772 -> 0.3561 | Hurt |
| use only ImageNet class labels as the concept set (ablation) | LE (SigLIP) · ResNet-50 (ImageNet), layer4 | correlation: 0.3772 -> 0.3378 | Hurt |
| replace SigLIP-ViT-L-16-384 explainer with CLIP ViT-L-14-336 (ablation) | LE (SigLIP) · ResNet-50 (ImageNet), layer4 | correlation: 0.3772 -> 0.3243 | Hurt |
| remove greedy search and use the top-5 concepts (ablation) | LE · ResNet-50 (ImageNet), layer4 | correlation: 0.2924 -> 0.3136; 0.3772 -> 0.3767 | Mixed |
| remove greedy search and use the top-4 concepts (ablation) | LE · ResNet-50 (ImageNet), layer4 | correlation: 0.2924 -> 0.2962; 0.3772 -> 0.3548 | Mixed |
| optimize the scaling parameters instead of using Norm scaling (ablation) | simulation ablation scoring · ResNet-50 (ImageNet), layer4 | ablation score: 0.0334 $\pm$ 0.0007 -> 0.0433 $\pm$ 0.0007; 0.0602 $\pm$ 0.0009 -> 0.0727 $\pm$ 0.0008 | Helped |
Surprising findings
- The highest 0.2% of activating inputs explain only about 1.5% of a ResNet-50 neuron's causal impact, suggesting top-activation explanations are insufficient.
- Simulation ablation scores for all existing neuron explanation methods are very low (at most 0.02), even though correlation scores are moderate.
Limitations
- Simulation scores remain low in absolute terms, especially ablation scores, so explanations are still far from faithful.
- Method works on image-level concepts and cannot capture location-specific activations, hurting lower-layer neurons.
- ViT-L/16 has all last-block MLP neurons dead, so it was excluded from main results.
Experimental setup
Datasets: ImageNet Places365 CIFAR-100 Broden WordNet
Benchmarks: ImageNet Places365 CIFAR-100
Metrics: Top Impact correlation score ablation score explanation length
Baselines: Network Dissection MILAN CLIP-Dissect
Builds on: Network Dissection MILAN CLIP-Dissect Compositional Explanations simulation scoring (Bills et al.)
Closest papers by meaning
- Evaluating Neuron Explanations: A Unified Framework with Sanity Checks (ICML 2025 · similarity 0.89)
- Decomposing stimulus-specific sensory neural information via diffusion models (NeurIPS 2025 · similarity 0.84)
- What should a neuron aim for? Designing local objective functions based on information theory (ICLR 2025 · similarity 0.84)
- Automated Natural Language Explanation of Deep Visual Neurons with Large Models (Student Abstract) (AAAI 2024 · similarity 0.84)
- Interpreting the Second-Order Effects of Neurons in CLIP (ICLR 2025 · similarity 0.84)
- NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.