Research map / Architectures and efficiency / RNNs and normalization
Adaptive Parametric Activation
Konstantinos P Alexandridis, Jiankang Deng, Anh Nguyen, Shan Luo
This paper tried a common change: Add dropout
add dropout to APA attention · ImageNet-LT · average top-1 accuracy: 57.0 -> 57.3 Improved
What 795 accepted papers report for this change
- 549 improved
- 87 no clear effect
- 12 mixed
- 147 hurt
- Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference ICML 2026 · apply layer dropout to whole transformer blocks instead of attention and FFN sub-blocks separately · 20 TPP val loss, 271M · val loss: 2.377 -> 2.367Improved
- Quantifying the Uncertainty of Foundation Models with Singular Value Ensembles ICML 2026 · use MC Dropout · vision and SST-2 · accuracy: 86.3 -> 76.6 on Flowers102Hurt
- Motion Planning in Compressed Representation Spaces ICML 2026 · add nested dropout during training · WOMD guided generation · goal FDE: 4.145 -> 1.104Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Sigmoid in channel attention with APA (ablation) | SE-ResNet50 · ImageNet-LT | average top-1 accuracy: 56.0 -> 57.0 | Improved |
| use APA instead of Gumbel with learnable temperature (ablation) | SE-ResNet50 · ImageNet-LT | average top-1 accuracy: 56.7 -> 57.0 | Improved |
| add learnable temperature to Sigmoid (ablation) | SE-ResNet50 · ImageNet-LT | Many-shot accuracy: Few 37.1 -> 40.3; Many 66.2 -> 65.9 | Mixed |
| add channel attention (SE) to plain ResNet50 (ablation) | ResNet50 · ImageNet-LT | average top-1 accuracy: 55.0 -> 56.0 | Improved |
| add dropout to APA attention (ablation) | SE + APA ResNet50 · ImageNet-LT | average top-1 accuracy: 57.0 -> 57.3 | Improved |
| add LayerNorm to APA attention (ablation) | SE + APA + Dropout ResNet50 · ImageNet-LT | average top-1 accuracy: 57.3 -> 57.4 | No clear effect |
| replace ReLU with AGLU in intermediate layers (ablation) | APA* ResNet50 · ImageNet-LT | average top-1 accuracy: 57.4 -> 57.9 | Improved |
| use PReLU or ELU instead of ReLU as intermediate activation (ablation) | APA* ResNet50 · ImageNet-LT | average top-1 accuracy: 57.4 -> 54.8 (PReLU), 52.6 (ELU) | Hurt |
| use APA* with deeper ResNet-101 (ablation) | SE-ResNet101 · ImageNet-LT | average top-1 accuracy: 56.7 -> 58.8 | Improved |
| add APA* to classifier learning methods such as cRT and BSCE (ablation) | ImageNet-LT classifier learning · ImageNet-LT | average top-1 accuracy: CE 51.7 -> 52.9; BSCE 56.0 -> 57.0 | Improved |
| add APA to Spatial attention (ablation) | Spatial attention ResNet50 · ImageNet-LT | average top-1 accuracy: 54.8 -> 55.2 | Improved |
| add APA*+AGLU to SE-ResNet50 on balanced ImageNet1K | SE-ResNet50 · ImageNet1K | top-1 accuracy: 77.5 -> 78.7 | Improved |
Closest papers by meaning
- Trainable Highly-expressive Activation Functions (ECCV 2024 · similarity 0.85)
- Efficient Activation Function Optimization through Surrogate Modeling (NeurIPS 2023 · similarity 0.84)
- Stochastic Adaptive Activation Function (NeurIPS 2022 · similarity 0.84)
- Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild (CVPR 2025 · similarity 0.84)
- LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs (AAAI 2025 · similarity 0.84)
- DiGRAF: Diffeomorphic Graph-Adaptive Activation Function (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
