Research map / Label-efficient and robust learning / Recognition and long-tail
Evaluating Neuron Explanations: A Unified Framework with Sanity Checks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies 18 neuron-explanation evaluation metrics under a single NeuronEval framework and proposes Missing/Extra Labels sanity tests, finding that F1, IoU, Correlation, Cosine and AUPRC pass while Recall, AUC, Accuracy and top-and-random sampling fail.
Problem
Neuron explanations are evaluated with many inconsistent metrics across papers, and it is unclear which metrics reliably measure whether a text explanation faithfully describes a unit's activations.
Approach
Write all existing evaluations as s_M(a_k, c_t), a metric applied to the binarized neuron activation vector and concept presence vector, then test each metric with Missing Labels (too-specific concept) and Extra Labels (too-generic concept) sanity checks, both empirically and theoretically.
Key results
- F1-score, IoU, Correlation, Cosine and AUPRC pass both Missing and Extra Labels tests (experimental and theoretical), while Recall fails the Extra Labels test with 0.00% Decrease Acc.
- Correlation achieves the top meta-AUPRC of 0.8765 and average rank 1.60 across 8 settings for identifying correct explanations on neurons with known concepts; Cosine (0.8666) and AUPRC (0.8406) follow.
- Correlation with top-and-random sampling fails the Extra Labels test (60.26% experimental, 43.64% theoretical Decrease Acc).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Correlation instead of the other evaluated metrics | Neuron explanation evaluation · Known-concept neurons (8 settings) | meta-AUPRC: 0.8765 | Helped |
| Use Cosine instead of the other evaluated metrics | Neuron explanation evaluation · Known-concept neurons (8 settings) | meta-AUPRC: 0.8666 | Helped |
| Use AUPRC instead of the other evaluated metrics | Neuron explanation evaluation · Known-concept neurons (8 settings) | meta-AUPRC: 0.8406 | Helped |
| Use F1-score or IoU instead of Recall | Neuron explanation evaluation · Missing Labels Test and Extra Labels Test | Decrease Acc: Recall: 98.66%, 0.00%; F1-score: 93.68%, 99.82% | Helped |
| Use uniform-sampling Correlation rather than top-and-random sampling | Neuron explanation evaluation · Extra Labels Test | Decrease Acc: 99.92% vs 60.26% | Helped |
| Use Precision rather than Recall | Neuron explanation evaluation · Missing Labels Test | Decrease Acc: 98.66% vs 45.73% | Hurt |
| Use the correct concept rather than a randomly label-doubled superset | Recall evaluation · Extra Labels Test | Decrease Acc: 0.00% | No clear effect |
| Use the correct concept rather than a randomly label-doubled superset | Accuracy evaluation · Extra Labels Test | Decrease Acc: 70.37% | Hurt |
| Use top-and-random sampling instead of uniform sampling | Correlation evaluation · Extra Labels Test | Decrease Acc: 99.92% -> 60.26% | Hurt |
| Use a metric that passes both sanity tests rather than a failing metric | Neuron explanation evaluation · Known-concept neurons (8 settings) | meta-AUPRC: Correlation 0.8765; Spearman Correlation 0.0853 | Helped |
Surprising findings
- Recall, the most common crowdsourced metric, gives a perfect score to overly generic concepts (e.g. 'animal' for a pets neuron) and fails the Extra Labels test completely.
- Top-and-random sampling makes Correlation fail the Extra Labels test, even though it is widely used in language neuron evaluation.
Limitations
- Passing the sanity tests is necessary but not sufficient for a reliable metric.
- Meta-evaluation on neurons with known concepts is mostly on final-layer or CBM neurons, which may not generalize to hidden-layer neurons.
- Framework covers input-based explanations only, not output-based ones.
Experimental setup
Datasets: ImageNet Places365 CUB200 OpenWebText
Benchmarks: ImageNet Places365 CUB200 OpenWebText
Metrics: Decrease Acc meta-AUPRC AUPRC
Baselines: Recall Precision IoU AUC Correlation (top-and-random) MAD WPMI Cosine
Builds on: Network Dissection TCAV Sanity checks for saliency maps
Closest papers by meaning
- Linear Explanations for Individual Neurons (ICML 2024 · similarity 0.89)
- WWW: A Unified Framework for Explaining What Where and Why of Neural Networks by Interpretation of Neuron Concepts (CVPR 2024 · similarity 0.88)
- NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions (ICLR 2025 · similarity 0.87)
- CoSy: Evaluating Textual Explanations of Neurons (NeurIPS 2024 · similarity 0.87)
- Neuron Explanations for Conformal Prediction (Student Abstract) (AAAI 2025 · similarity 0.86)
- Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.