Research map / Label-efficient and robust learning / Recognition and long-tail
Don't trust your eyes: on the (un)reliability of feature visualizations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows feature visualizations can be manipulated via fooling circuits and silent units, that last-layer visualizations are processed along different paths than natural images, and proves that argmax-based feature visualization cannot reliably predict black-box functions.
Problem
Whether feature visualizations (activation maximization) are reliable for understanding neural network units, both under adversarial model manipulation and in standard non-adversarial networks.
Approach
Builds network circuits (a fooling circuit with a binary natural-vs-visualization classifier, and silent-unit orthogonal filters) that keep natural-input behavior while making feature visualizations show arbitrary patterns; proposes a similarity-based sanity check comparing activations of visualizations and natural images; and proves impossibility results showing argmax/argmin-based visualizations
Key results
- Fooling circuit on Inception-V1 last layer changes top-1 accuracy only from 69.146% to 68.744% while visualizations are arbitrarily permuted or replaced by a Mona Lisa pattern.
- Silent-unit method makes all 512 units in a ResNet-50 layer yield near-identical visualizations with no change in top-1 or top-5 validation accuracy.
- Sanity check shows that in most early and mid layers of Inception-V1, same-class natural image similarity to last-layer visualizations is about as low as similarity to other-class images; theory shows no argmax-based decoder can reliably predict black-box or NN functions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a fooling circuit to manipulate last-layer feature visualizations | Inception-V1 · ImageNet-1K validation | top-1 accuracy: 69.146\,\% -> 68.744\,\% | Hurt |
| Add a fooling circuit to manipulate last-layer feature visualizations | Inception-V1 · ImageNet-1K validation | top-5 accuracy: 88.858\,\% -> 88.330\,\% | Hurt |
| Add fooling circuits to decouple feature visualizations from behavior on natural input | Inception-V1 · ImageNet-1K validation | top-1 accuracy: 69.146\,\% -> 68.744\,% | No clear effect |
| Add orthogonal filters in silent units so layer visualizations become near-identical | ResNet-50 · ImageNet-1K validation | top-1 and top-5 accuracy | No clear effect |
| Use feature visualizations instead of natural images as inputs | Inception-V1 feature visualization sanity check · Inception-V1 layers | activation similarity | Hurt |
Surprising findings
- Feature visualizations of a unit are processed along paths largely different from natural images of the same class in most layers of Inception-V1.
- Even convex and Lipschitz function classes admit impossibility results for predicting f(x) from argmax/argmin feature visualizations.
Limitations
- Fooling demonstrations assume white-box access to the model and a fixed visualization method, a threat scenario that may be rare in practice.
- Sanity check is only a necessary, not sufficient, condition and was run on a single method and single network (Inception-V1 last layer).
- Theoretical results are proven for 1-dimensional (or componentwise) functions and bounded inputs, with some extensions only sketched.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy top-5 accuracy Spearman rank correlation similarity validation loss
Baselines: feature visualization (Olah et al. 2017)
Builds on: activation maximization feature visualization Inception-V1 / GoogLeNet
Closest papers by meaning
- Understanding Visual Feature Reliance through the Lens of Complexity (NeurIPS 2024 · similarity 0.86)
- Manipulating Feature Visualizations with Gradient Slingshots (NeurIPS 2025 · similarity 0.84)
- Neuro-Visualizer: A Novel Auto-Encoder-Based Loss Landscape Visualization Method With an Application in Knowledge-Guided Machine Learning (ICML 2024 · similarity 0.83)
- DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations (CVPR 2024 · similarity 0.83)
- Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory (ICLR 2024 · similarity 0.83)
- Parallel Backpropagation for Shared-Feature Visualization (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.