Research map / Architectures and efficiency / CNN design
Targeted Activation Penalties Help CNNs Ignore Spurious Signals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Targeted Activation Penalty (TAP), which penalises CNN activations at spurious-signal locations (derived from ground-truth or teacher-generated masks) to stop spurious signals re-emerging in deeper layers, with lower training cost than RRR/RBR.
Problem
Deep CNNs trained with explanatory supervision (RRR, RBR) still rely on spurious signals that re-emerge in deeper layers, and those methods require costly ground-truth annotations and expensive second-order derivatives.
Approach
Penalise the L1 norm of channel-summed activations at spatial locations corresponding to spurious-signal masks across selected conv layers, which indirectly suppresses input gradients without computing second-order derivatives; masks can come from a teacher model's saliency maps (pre-trained on 1% or 10% clean data) instead of humans.
Key results
- TAP with ground-truth annotations matches Base on PNEU across DenseNet-121, ResNet-18 and VGG-16 with little contamination sensitivity, whereas RRR and RBR show large drops on some architectures (e.g., RRR on DenseNet-121).
- TAP trains at about 11.8-36.7 it/s versus 1.7-12.5 for RRR/RBR and uses roughly half the memory on ResNet-18 (2.5 GB vs 3.2-4.0 GB), while RBR runs out of memory on DenseNet-121 at batch size 16.
- With teacher annotations on PNEU, TAP students outperform the 10%-teacher (F-score 0.869) and 1%-teacher (F-score 0.668), while random annotations rely on spurious correlations for two of three models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TAP instead of RRR or RBR with ground-truth annotations | CNNs on PNEU and KNEE · PNEU and KNEE contaminated and permuted test sets | contamination sensitivity | Helped |
| Use TAP instead of RRR or RBR | Two-layer CNN on MNIST · MNIST | accuracy: 0.987 (0.001) -> 0.983 (0.001) | Mixed |
| Use teacher-generated masks instead of random annotations with TAP | TAP student models · PNEU | contamination sensitivity | Helped |
| Use teacher-generated masks with TAP rather than ground-truth masks | TAP student models · KNEE | MAE | Mixed |
| Use a 1%-clean-data teacher rather than a 10%-clean-data teacher | TAP student models · PNEU | F-score | No clear effect |
| Increase the teacher-mask threshold on KNEE | TAP with teacher annotations · KNEE | contamination sensitivity | Hurt |
| Use TAP rather than RRR or RBR | CNNs on PNEU with stripe artifacts · PNEU permuted test | F-score | Mixed |
| Use TAP rather than RRR or RBR | CNNs on KNEE with stripe artifacts · KNEE permuted test | MAE | Mixed |
| Use TAP rather than RRR | VGG-16 training · 224×224 medical-image batches | iterations per second: 2.6 -> 11.8 | Helped |
| Use TAP rather than RBR | DenseNet-121 training · 224×224 medical-image batches | iterations per second: OOM -> 12.0 | Helped |
Surprising findings
- On MNIST with a shallow two-layer CNN, TAP is slightly outperformed by RRR and RBR, which the authors attribute to TAP being specialised for deep CNNs.
- Increasing the threshold tau for teacher annotations surprisingly increased contamination sensitivity for DenseNet-121 and VGG-16 on KNEE and PNEU.
Limitations
- Method is specialised to CNNs; the authors plan to test generalisation to other architectures.
- Teacher annotations are noisier than ground truth and do not work uniformly across tasks, models and thresholds (e.g., ResNet-18 trained poorly on KNEE with teacher masks).
- Evaluation uses synthetic spurious contaminations on MNIST and two medical datasets, a small set of architectures and a single resolution.
Experimental setup
Datasets: MNIST PNEU (paediatric chest radiographs) KNEE (osteoarthritis knee radiographs)
Benchmarks: MNIST PNEU KNEE
Metrics: accuracy F-score mean absolute error delta-metric under permuted contamination precision/recall of top-quartile input gradients and activat iterations per second memory usage
Baselines: RRR RBR No XS Base
Builds on: right for the right reasons (RRR) right for better reasons (RBR) knowledge distillation
Closest papers by meaning
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.85)
- TAPT: Test-Time Adversarial Prompt Tuning for Robust Inference in Vision-Language Models (CVPR 2025 · similarity 0.83)
- NeuronTune: Towards Self-Guided Spurious Bias Mitigation (ICML 2025 · similarity 0.83)
- Severing Spurious Correlations with Data Pruning (ICLR 2025 · similarity 0.82)
- Attacking CNNs in Histopathology with SNAP: Sporadic and Naturalistic Adversarial Patches (Student Abstract) (AAAI 2024 · similarity 0.82)
- How Spurious Features are Memorized: Precise Analysis for Random and NTK Features (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.