Research map / Architectures and efficiency / RNNs and normalization
Manipulating Feature Visualizations with Gradient Slingshots
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Gradient Slingshots, a fine-tuning attack that makes feature visualization (FV) of a neuron converge to an arbitrary target image while preserving architecture, accuracy and feature function, and proposes an activation-maximization-based detection defense.
Problem
Feature Visualization is used to audit DNN features, but its trustworthiness is unclear; it is unknown whether FV outputs can be covertly changed without altering architecture or significantly degrading performance.
Approach
Fine-tune the target feature so its gradient field points toward a chosen target point within a tunnel connecting the FV initialization region to a landing zone around the target, plus a preservation loss on internal activations, so standard gradient-ascent FV converges to the target while the model's behavior is kept.
Key results
- On ResNet-50 wolf spider neuron, alpha=0.64 yields target-label alignment 0.31 vs 0.23 original, GT alignment 0.23 vs 0.29, with accuracy 75.13% vs 76.13% and AUROC 1.00 preserved.
- In the CLIP ViT-L/14 weapon-detection case study, manipulated FV CLIP similarity to gentoo penguin target rises from 0.60 to 0.88 while AUROC for assault rifle stays at 1.00.
- GS outperforms ProxPulse on ResNet-50 at comparable settings, e.g., ProxPulse alpha=0.1 drops accuracy to 58.51% vs 75.13% for GS.
- Top-100 natural AM signals overlap with Jaccard similarity 0.84 (assault rifle) and 0.56 (wolf spider), enabling detection of manipulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the wolf spider feature toward a Dalmatian target with Gradient Slingshots (α=0.64) rather than leave the model unchanged | ResNet-50 · Wolf spider FV manipulation | CLIP target similarity: 0.53±0.02 -> 0.69±0.02 | Helped |
| Use Gradient Slingshots rather than ProxPulse | ResNet-50 · Wolf spider FV manipulation | test accuracy (%): 69.03 -> 75.13 | Helped |
| Use Gradient Slingshots rather than ProxPulse | ResNet-50 · Wolf spider FV manipulation | CLIP target similarity: 0.53±0.01 -> 0.69±0.02 | Helped |
| Set preservation-loss weight α to 0.90 rather than 0.64 (ablation) | Gradient Slingshots · ResNet-50 wolf spider FV manipulation | CLIP target similarity: 0.69±0.02 -> 0.53±0.02 | Hurt |
| Set preservation-loss weight α to 0.50 rather than 0.64 (ablation) | Gradient Slingshots · ResNet-50 wolf spider FV manipulation | test accuracy (%): 74.83 -> 71.52 | Hurt |
| Set preservation-loss weight α to 0.10 rather than 0.64 (ablation) | Gradient Slingshots · ResNet-50 wolf spider FV manipulation | CLIP target similarity: 0.69±0.02 -> 0.59±0.02 | Hurt |
| Increase VGG width from ×8 to ×64 (ablation) | Gradient Slingshots · CIFAR-10 cat feature FV to goldfish target | LPIPS: 0.17±0.02 -> 0.03±0.01 | Helped |
| Increase model depth at fixed width (ablation) | Gradient Slingshots · CIFAR-10 cat feature FV to goldfish target | LPIPS | Mixed |
| Choose an in-distribution target image rather than an out-of-distribution target (ablation) | Gradient Slingshots · TinyImageNet ResNet-18 gondola feature | LPIPS | Helped |
| Use a training sample rather than a hold-out sample as the target (ablation) | Gradient Slingshots · TinyImageNet ResNet-18 gondola feature | LPIPS | No clear effect |
| Manipulate the assault rifle feature toward a gentoo penguin target | CLIP ViT-L/14 weapon-detection model · Assault rifle feature FV | CLIP target similarity: 0.60 ± 0.04 -> 0.88 ± 0.02 | Helped |
| Use natural-domain top activating samples to detect manipulation | Gradient Slingshots · Assault rifle and wolf spider features | Jaccard similarity: 0.84 for assault rifle; 0.56 for wolf spider | Helped |
Surprising findings
- Manipulation success increases with model size, with the widest models giving best manipulation, while deeper models are not always closer to target due to shattered gradients.
- Very low alpha values also lead to poor manipulation success, and accuracy drops sharply at high manipulation weight (e.g., alpha=0.20 gives 21.97% accuracy).
Limitations
- Selecting GS hyperparameters (alpha grid search) incurs computational overhead.
- Robustness to all FV techniques and regularization strategies not exhaustively tested.
- Detection method relies on labeled test data and may degrade on out-of-distribution inputs.
Experimental setup
Datasets: MNIST CIFAR-10 TinyImageNet ImageNet-1k Weapon Detection Dataset
Benchmarks: MNIST CIFAR-10 TinyImageNet ImageNet-1k
Metrics: AUROC classification accuracy CLIP target label alignment CLIP ground-truth label alignment CLIP image similarity LPIPS MSE SSIM Jaccard similarity
Baselines: ProxPulse fooling circuits silent units
Builds on: Feature Visualization fooling circuits ProxPulse physics-informed neural networks lucid
Closest papers by meaning
- Don't trust your eyes: on the (un)reliability of feature visualizations (ICML 2024 · similarity 0.84)
- ADAPT: Hybrid Prompt Optimization for LLM Feature Visualization (NeurIPS 2026 · similarity 0.84)
- Adversarial Attacks on the Interpretation of Neuron Activation Maximization (AAAI 2024 · similarity 0.83)
- Understanding Visual Feature Reliance through the Lens of Complexity (NeurIPS 2024 · similarity 0.83)
- FastCAV: Efficient Computation of Concept Activation Vectors for Explaining Deep Neural Networks (ICML 2025 · similarity 0.83)
- DiffCAM: Data-Driven Saliency Maps by Capturing Feature Differences (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.