Research map / Architectures and efficiency / Attention and transformers
EAN: An Efficient Attention Module Guided by Normalization for Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EAN, a lightweight plug-and-play attention module that reuses GroupNorm affine scaling factors (after L2 normalization and a learned adaptation) as channel attention weights, avoiding fully-connected layers.
Problem
Channel attention modules such as SE rely on costly fully-connected layers, and existing combinations of normalization and attention treat them as separate or simplistically combined components.
Approach
Grouping channels into sub-features and using the GroupNorm affine scale factors, passed through L2 normalization and a learnable scale-and-shift with sigmoid, as the attention mask, so normalization and attention share one parameterization with almost no extra parameters.
Key results
- On ImageNet-1K, EAN improves MobileNeXt, MobileNetV2 and ResNet-50 top-1 by 2.42%, 1.78% and 1.86% with almost no extra parameters (ResNet-50 79.45% vs 77.60% baseline).
- On MS COCO 2017, EAN outperforms CA by 1.1% and 2.3% on the two detection frameworks, with EAN-ResNet50 reaching 42.2 mAP versus 41.7 baseline.
- On CIFAR-100 with MobileNeXt, EAN reaches 60.88% vs 54.77% for the baseline and 58.58% for the best competing attention (CA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EAN to MobileNetV2 | MobileNetV2 · CIFAR-10 | accuracy: 85.89 -> 86.74 | Helped |
| Add EAN to MobileNetV2 | MobileNetV2 · CIFAR-100 | accuracy: 55.85 -> 56.20 | Helped |
| Add EAN to MobileNeXt | MobileNeXt · CIFAR-100 | accuracy: 58.58 -> 60.88 | Helped |
| Add EAN to ResNet50 | ResNet50 · ImageNet-1K | Top-1 accuracy: 77.40 -> 77.87 | Helped |
| Add EAN to MobileNeXt with SSDLite320 | MobileNeXt · PASCAL VOC 2012 val | mAP: 72.9 -> 73.4 | Helped |
| Add EAN to MobileNeXt | MobileNeXt · MS COCO 2017 val | mAP: 23.0 -> 23.5 | Helped |
| Add EAN to ResNet50 | ResNet50 · MS COCO 2017 val | mAP: 38.0 -> 38.4 | Helped |
| Vary the grouping number G (ablation) | EAN with ResNet-50 and MobileNeXt · CIFAR-100 | Top-1 error | Mixed |
Limitations
- Placement of the module within each architecture was chosen per backbone and is noted as critical, with no general rule.
- Grouping number G must be tuned (recommended 8 for lightweight and 64 for larger models), based on limited ablation on CIFAR-100.
- Evaluation is limited to CNN backbones and visual tasks; no transformer or language experiments.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 PASCAL VOC MS COCO
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 PASCAL VOC 2012 MS COCO 2017
Metrics: top-1 accuracy mAP AP@.5 AP@.75 params FLOPs FPS
Baselines: SE CBAM ECA SA SGE CA PdfAM SE-MobileNeXt CA-MobileNeXt
Builds on: Group Normalization Squeeze-and-Excitation Attentive Normalization SGE Coordinate Attention
Closest papers by meaning
- SimAM: A Simple, Parameter-Free Attention Module for Convolutional Neural Networks (ICML 2021 · similarity 0.88)
- DIANet: Dense-and-Implicit Attention Network (AAAI 2020 · similarity 0.86)
- Bayesian Attention Modules (NeurIPS 2020 · similarity 0.85)
- A Primal-Dual Framework for Transformers and Neural Networks (ICLR 2023 · similarity 0.85)
- Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiency (NeurIPS 2025 · similarity 0.84)
- STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.