Research map / Architectures and efficiency / RNNs and normalization
Measuring Per-Unit Interpretability at Scale Without Humans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Machine Interpretability Score (MIS), a fully automated per-unit interpretability metric for vision DNNs that replaces human 2-AFC psychophysics with DreamSim similarity, and uses it to analyze 835 models and training dynamics.
Problem
Measuring per-unit interpretability of vision DNNs currently requires costly human psychophysics experiments, which does not scale and limits interpretability research.
Approach
Replace human participants in the 2-AFC psychophysics task with a machine classifier that matches query images to positive/negative visual explanations using perceptual similarity (DreamSim), yielding a human-free score (MIS) with a single calibration parameter alpha.
Key results
- MIS reproduces model interpretability rankings from human scores with Pearson r = 0.98 and Spearman r = 0.94 across nine models.
- Per-unit MIS correlates with human interpretability scores (Pearson and Spearman 0.80), matching a noise ceiling of about 0.82; on new units selected by MIS, a psychophysics study with 236 participants gave Pearson 0.85.
- Across 835 models, average per-unit MIS is anticorrelated with ImageNet accuracy (Pearson r = -0.5, Spearman r = -0.55).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DreamSim to compute image similarity | MIS on IMI per-unit annotations · IMI | correlation with HIS: 0.80 Pearson; 0.80 Spearman | Helped |
| replace DreamSim with LPIPS (ablation) | MIS · IMI per-unit | correlation with HIS: 0.80 -> 0.70 Pearson; 0.80 -> 0.70 Spearman | Hurt |
| replace DreamSim with DISTS (ablation) | MIS · IMI per-unit | correlation with HIS: 0.80 -> 0.95 Pearson; 0.80 -> 0.70 Spearman | Mixed |
| use feature visualizations instead of dataset examples (ablation) | MIS · IMI model-level | correlation with HIS: 0.95 Pearson; 0.88 Spearman | No clear effect |
| use the MIS to select easiest and hardest units before human testing | GoogLeNet and ResNet-50 · newly tested units | human interpretability: easiest MIS 0.99; hardest MIS 0.63 and 0.59 | Helped |
| increase classification performance | vision models · ImageNet-2012 | average per-unit MIS: Pearson’s r = −0.5; Spearman’s r = −0.55 | Hurt |
| use LPIPS or DISTS instead of DreamSim at the model level (ablation) | MIS · IMI model-level | correlation with HIS: LPIPS: ρs = 0.95, ρp = 0.98; DISTS: ρs = 0.96, ρp = 0.99 | No clear effect |
| replace a layer with a standard SAE or Gated SAE (ablation) | GoogLeNet mixed4b_3x3 · SAE interpretability | MIS: 91.21 % | No clear effect |
| increase SAE sparsity weight across the tested settings (ablation) | ResNet-50 layer2_2_conv2 SAE · SAE hyperparameter sensitivity | MIS: 0.892; 0.908; 0.916; 0.915; 0.919; 0.918; 0.922; 0.918 | Mixed |
| increase number of MIS tasks (ablation) | MIS · GoogLeNet | absolute difference from MIS using 20 tasks | Helped |
| train ResNet-50 beyond its first epoch | ResNet-50 · ImageNet-2012 training | MIS: MIS increases to around 0.93 after the first epoch, before it decays | Hurt |
| increase relative layer depth | Conv/Linear, BatchNorm, and LayerNorm layers · 835 vision models | average per-unit MIS: Conv/Linear ρs = 0.85, ρp = 0.85; BatchNorm ρs = 0.77, ρp = 0.77; LayerNorm ρs = 0.85, ρp = 0.86 | Mixed |
Surprising findings
- Better ImageNet classifiers tend to have less interpretable units, contrary to the expectation that interpretability and accuracy go together.
- MIS of an untrained ResNet-50 is already above chance and rises sharply in the first training epoch before decaying as accuracy grows.
Limitations
- Per-unit MIS performance is close to the noise ceiling of the limited human IMI annotations, so finer improvements are hard to validate.
- MIS depends on the explanation method and the single calibration parameter alpha was fit on IMI annotations.
- Only validated on computer vision models and units of channels/neurons; the analysis of SAEs found no systematic benefit.
Experimental setup
Datasets: ImageNet-2012 IMI (ImageNet Mechanistic Interpretability) dataset
Benchmarks: IMI human interpretability annotations ImageNet-2012 top-1 accuracy
Metrics: Pearson correlation Spearman correlation average per-unit MIS psychophysics accuracy
Baselines: LPIPS DISTS Human interpretability scores (IMI)
Builds on: Borowski et al. 2-AFC psychophysics task IMI dataset (Zimmermann et al. 2023) DreamSim
Closest papers by meaning
- Scale Alone Does not Improve Mechanistic Interpretability in Vision Models (NeurIPS 2023 · similarity 0.91)
- Enhancing Pre-trained Representation Classifiability can Boost its Interpretability (ICLR 2025 · similarity 0.87)
- A Multimodal Automated Interpretability Agent (ICML 2024 · similarity 0.87)
- Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment (AAAI 2025 · similarity 0.87)
- Characterizing Universal Object Representations Across Vision Models (NeurIPS 2026 · similarity 0.87)
- Intrinsic User-Centric Interpretability through Global Mixture of Experts (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.