Research map / Architectures and efficiency / CNN design
AdaShift: Learning Discriminative Self-Gated Neural Feature Activation With an Adaptive Shift Factor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AdaShift, a self-gated activation function that adds an adaptive shift factor (computed via LayerNorm over channel or patch statistics) to the input of the sigmoid re-weighting, improving ResNet/CIFAR-ResNet accuracy on ImageNet and CIFAR-100 over ReLU and SOTA activations.
Problem
Self-gated activation functions have fixed re-weighting curves that cannot adapt well to training conditions, and norm-induced mismatched feature scoring limits their discriminativeness; prior SOTA (IIEU) addresses this with costly norm-decoupling.
Approach
Reformulate self-gated activation as sigmoid(x + Δ)·x, where the shift Δ is a cheap LayerNorm-based function of tensor-level channel or local-patch statistics (AdaShift-B and AdaShift-MA), giving dynamic, norm-aware re-weighting with negligible overhead and no norm decoupling.
Key results
- On ImageNet with ResNet-50, AdaShift-MA reaches 80.3% Top-1 vs. 77.2% for ReLU (+3.1) and 79.7% for IIEU, with ~0.1M extra parameters.
- On ImageNet with ResNet-14, AdaShift-MA reaches 73.9% vs. 68.7% for ReLU (+5.2), and ResNet-101 with AdaShift-MA reaches 81.2% vs. 78.9% for ReLU.
- On CIFAR-100 with CF-ResNet-56, AdaShift-B reaches 76.5% and AdaShift-MA 77.0% (mean of 8 runs) vs. 74.4% for ReLU and 75.3% for SiLU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ReLU with AdaShift-MA | ResNet-50 · ImageNet | Top-1 accuracy (%): 77.2 -> 80.3 | Helped |
| replace ReLU with AdaShift-MA | ResNet-101 · ImageNet | Top-1 accuracy (%): 78.9 -> 81.2 | Helped |
| replace ReLU with AdaShift-MA | CF-ResNet-56 · CIFAR-100 | Top-1 accuracy: 74.4˘0.3 -> 77.0˘0.4 | Helped |
| add a shift inside the reweighting function while leaving the multiplied feature unshifted (ablation) | AdaShift-B · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 75.3˘0.4 -> 76.5˘0.3 | Helped |
| scale the activation input with a trainable κ (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 74.8˘0.2 | Hurt |
| use Δ to multiply the activation input before reweighting (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 73.4˘0.3 | Hurt |
| combine trainable input scaling κ with shift Δ (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 73.7˘0.3 | Hurt |
| use two adaptive shifts, Δ1 and Δ2, in the reweighting input (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 73.6˘0.2 | Hurt |
| also shift the multiplied feature to x + Δ (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 75.9˘0.3 | Hurt |
| apply separate shifts inside reweighting and to the multiplied feature (ablation) | AdaShift prototype · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 vs 76.2˘0.4 | Hurt |
| remove Z-scoring from the shift by replacing LayerNorm with γx + β (ablation) | AdaShift-B · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 -> 76.0˘0.4 | Hurt |
| replace LayerNorm in the shift with a linear layer (ablation) | AdaShift-B · CIFAR-100, CF-ResNet-56 | Top-1 accuracy: 76.5˘0.3 -> 76.3˘0.2 | Hurt |
Surprising findings
- Directly adding the shift to the input outside the re-weighting function (CG6-style) improves over SiLU but underperforms AdaShift, and stacking two shift paths yields no gain.
- Removing the Z-scoring of LayerNorm or replacing LN with a linear layer in the shift factor degrades accuracy, indicating balanced summation of the input and shift matters.
Limitations
- Gains on large-scale benchmarks are reported on ResNets and lightweight CNNs; applicability to Transformer FFNs requires a tailored variant described only in the supplement.
- Formulation of the shift assumes normalized inputs (BN/LN), requiring modifications for unnormalized activations.
Experimental setup
Datasets: ImageNet CIFAR-100 COCO KITTI-Materials
Benchmarks: ImageNet CIFAR-100 COCO KITTI-Materials
Metrics: Top-1 accuracy parameters FLOPs throughput
Baselines: ReLU Softplus LeakyReLU PReLU ELU GELU SiLU Mish Swish ACON-C Meta-ACON SMU
Builds on: SiLU Swish ACON IIEU Meta-ACON
Closest papers by meaning
- ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · similarity 0.84)
- Towards More Discriminative Feature Learning in SNNs with Temporal-Self-Erasing Supervision (AAAI 2025 · similarity 0.84)
- Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning (NeurIPS 2026 · similarity 0.84)
- Rethinking Neural Nonlinearity as Gating (NeurIPS 2026 · similarity 0.84)
- Rethinking Few-Shot Adaptation of Vision-Language Models in Two Stages (CVPR 2025 · similarity 0.84)
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.