Research map / Adversarial robustness and security / Attacks on text and graphs
Adversarial Attacks on the Interpretation of Neuron Activation Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adversarial model manipulation that fine-tunes a pretrained CNN so that feature-visualization top-k images for neurons/channels change (push-down, push-up, fairwashing) while preserving accuracy.
Problem
Feature visualization via activation maximization (top-k images) is popular for interpreting individual neurons, but it may be deceived by a model creator who manipulates the network before release.
Approach
Fine-tune a pretrained model with an attack loss on top-k images (push-down/push-up) or on moment-matching of pre-activations across protected groups (fairwashing), combined with a distillation loss that keeps outputs close to the original model, and propose whack-a-mole and CLIP-based metrics to quantify success.
Key results
- Push-down all-channel attack on AlexNet Conv5 replaces top-5 images for visualized channels with final validation accuracy 56.2% (less than half a percent drop).
- Fairwashing attack roughly doubles the balance measure for a biased unit (unit 800) while keeping accuracy (56.45 to 56.56) and DDI/DEO nearly unchanged.
- Synthetic feature visualizations are largely unchanged after natural top-k attacks, suggesting decorrelation between the two visualization types.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply all-channel Push-Down to AlexNet Conv5 | AlexNet · ImageNet, Conv5 top-k feature visualization | CLIP-δ: 0.249 | Helped |
| Apply all-channel Push-Down to AlexNet Conv5 | AlexNet · ImageNet val | accuracy: 56.2% | No clear effect |
| Apply all-channel Push-Up with Goldfish decoys to AlexNet Conv5 | AlexNet · ImageNet, Conv5 top-k feature visualization | CLIP-δ: 0.150 | Helped |
| Apply all-channel Push-Up with Goldfish decoys to AlexNet Conv5 | AlexNet · ImageNet val | accuracy: 56.3% | No clear effect |
| Apply all-channel Push-Down to AlexNet Conv1–Conv5 (ablation) | AlexNet · ImageNet, top-k feature visualization by layer | CLIP-δ: 0.043; 0.056; 0.127; 0.205; 0.249 | Mixed |
| Apply Push-Down to EfficientNet Feature 7 block 3 (ablation) | EfficientNet · ImageNet, Feature 7 block 3 top-k feature visualization | CLIP-δ: 0.262 | Helped |
| Replace natural top-k images with synthetic activation-maximizing images as the visualization (ablation) | AlexNet Conv5 Push-Down and Push-Up · Synthetic feature visualization | change in synthetic optimal image | No clear effect |
| Apply fairwashing fine-tuning to 200 human-related neurons | AlexNet · ImageNet People Subtree, gender-conditioned top-30 visualization | balance | Helped |
| Apply fairwashing fine-tuning to 200 human-related neurons | AlexNet · ImageNet val | accuracy: 56.45 -> 56.56 | No clear effect |
| Apply fairwashing fine-tuning to 200 human-related neurons | AlexNet · ImageNet People Subtree, Baseball player | DDI: 3.38 -> 3.14 | No clear effect |
| Apply fairwashing fine-tuning to 200 human-related neurons | AlexNet · ImageNet People Subtree, Bridegroom | DEO: 12.34 ->12.34 | No clear effect |
| Mask AlexNet channel 0 by setting all its weights to zero (ablation) | AlexNet · ImageNet | accuracy drop: 0.2% | Hurt |
Surprising findings
- Synthetic (activation-maximizing) feature visualizations remain largely unaffected even when natural top-k images are completely replaced.
- Setting a single channel's weights to zero achieves near-trivial push-down with only 0.2% accuracy loss, so single-channel attacks are not very constrained.
Limitations
- Whack-a-mole metrics may be imperfect, and validating them requires inspecting all channels.
- Fairwashing is restricted to binary protected attributes.
- Experiments are mainly on AlexNet; EfficientNet ablation is limited to one layer.
Experimental setup
Datasets: ImageNet-1k ImageNet People Subtree
Benchmarks: ImageNet validation set
Metrics: top-1 accuracy Kendall-tau CLIP-delta Kendall-tau-W CLIP-W balance KS distance DDI DEO
Baselines: Unmodified pretrained AlexNet
Builds on: activation maximization feature visualization Dimension-wise distillation (CKA-based loss, Davari et al.) MILAN
Closest papers by meaning
- Attack to Explain Deep Representation (CVPR 2020 · similarity 0.86)
- Synthesizing the preferred inputs for neurons in neural networks via deep generator networks (NeurIPS 2016 · similarity 0.86)
- An Adversarial Framework for Generating Unseen Images by Activation Maximization (AAAI 2022 · similarity 0.85)
- Improving Adversarial Transferability via Neuron Attribution-Based Attacks (CVPR 2022 · similarity 0.84)
- Manipulating Feature Visualizations with Gradient Slingshots (NeurIPS 2025 · similarity 0.83)
- Boosting Adversarial Transferability through Augmentation in Hypothesis Space (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.