Research map / Architectures and efficiency / CNN design
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add activation-based attention transfer | CIFAR-10 student CNNs · CIFAR-10 | test error: 9.38 -> 8.93; 8.77 -> 7.93; 8.77 -> 8.25; 6.31 -> 5.85 | Helped |
| Combine attention transfer with knowledge distillation | CIFAR-10 student CNNs · CIFAR-10 | test error: 8.33; 7.51; 8.01; 5.71 | Mixed |
| Transfer full activations instead of attention maps (ablation) | CIFAR-10 student CNNs · CIFAR-10 | test error: 8.93 -> 9.05; 7.93 -> 8.51; 8.25 -> 8.62; 5.85 -> 6.24 | Hurt |
| Use one attention-transfer loss at a time instead of one per group (ablation) | WRN-16-1 student with WRN-16-2 teacher · CIFAR-10 | test error: 7.93; 8.11; 7.96; 7.97 | Hurt |
| Use F_sum attention mapping rather than no attention transfer (ablation) | WRN-16-2/WRN-16-1 · CIFAR-10 | test error: 8.77 -> 7.99 | Helped |
| Use F^2_sum attention mapping rather than no attention transfer (ablation) | WRN-16-2/WRN-16-1 · CIFAR-10 | test error: 8.77 -> 7.93 | Helped |
| Use F^4_sum attention mapping rather than F^2_sum (ablation) | WRN-16-2/WRN-16-1 · CIFAR-10 | test error: 7.93 -> 8.09 | Hurt |
| Use max-based attention mapping rather than F^2_sum (ablation) | WRN-16-2/WRN-16-1 · CIFAR-10 | test error: 7.93 -> 8.08 | Hurt |
| Regularize with input-gradient l2 norm (ablation) | Thin NIN · CIFAR-10 | test error: 13.5 -> 12.5 | Helped |
| Transfer gradient-based attention | Thin NIN · CIFAR-10 | test error: 13.5 -> 12.1 | Helped |
| Enforce flip invariance on gradient attention maps (ablation) | Thin NIN · CIFAR-10 | test error: 13.5 -> 11.8 | Helped |
| Add activation-based attention transfer | Thin NIN · CIFAR-10 | test error: 13.5 -> 11.2 | Helped |
Closest papers by meaning
- Learn to Pay Attention (ICLR 2018 · similarity 0.90)
- SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks (ICML 2021 · similarity 0.88)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.88)
- Learning what and where to attend (ICLR 2019 · similarity 0.88)
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.