Research map / Architectures and efficiency / CNN design

LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs

AAAI 2025 · Architectures and efficiency › CNN design · approach: Theory of convolutional inductive bias · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LayerAct, a layer-level activation mechanism whose gating uses layer-normalized inputs (LA-SiLU, LA-HardSiLU), claimed to be more noise-robust and zero-like-mean than element-level activations, and evaluated on MNIST, CIFAR10/100, and ImageNet with ResNets.

Problem

Element-level activation functions trade off one-sided saturation against zero-like mean outputs and show sample-dependent noise robustness in CNNs, while LayerNorm-based activations homogenize outputs.

Approach

The activation scale is computed from the layer-normalized input (s(n_i)) while the output multiplies the raw input y_i, so the saturation threshold adapts per sample to the layer mean and variance, giving lower activation fluctuation under noise and no saturation-vs-negative-output trade-off.

Key results

• On noisy CIFAR10 with ResNet44, LA-HardSiLU reaches 32.27% on Gaussian Noise 3 versus 29.95% for ReLU, and LA-SiLU reaches 83.92% on Poisson noise versus 73.71% for ReLU.
• On clean ImageNet with ResNet50, LA-SiLU achieves 78.62% versus 77.85% for SiLU and 77.71% for ReLU, and LA-SiLU reaches 43.07% under Poisson noise versus 9.52% for SiLU.
• LA-SiLU outperforms element-level baselines in 30 of 36 significance comparisons on clean CIFAR10/CIFAR100 networks.

Measured effects reported in the paper

Surprising findings

• Element-level activations such as ReLU and LReLU sometimes beat LayerAct under mild Gaussian noise on CIFAR10 (e.g., ReLU 69.74 vs LA-SiLU 68.45 on Gaussian Noise 1), and LA-HardSiLU underperforms on clean CIFAR10/100.
• LayerAct gains are largest under Poisson noise, where baselines collapse (e.g., SiLU 7.15% vs LA-SiLU 49.33% on ImageNet).

Limitations

• Only unbounded LayerAct functions (LA-SiLU, LA-HardSiLU) are proposed; bounded variants for RNNs are left to future work.
• Noise-robustness gains are not uniform: mild Gaussian noise and clean accuracy sometimes favor element-level functions.
• Theoretical bounds rely on assumptions such as sigma_noise much smaller than sigma_y and large alpha.

Experimental setup

Datasets: MNIST CIFAR10 CIFAR100 ImageNet

Benchmarks: CIFAR10 CIFAR100 ImageNet MNIST

Metrics: accuracy standard deviation of accuracy activation fluctuation

Baselines: ReLU LReLU PReLU Mish SiLU HardSiLU

Builds on: SiLU HardSiLU LayerNorm ResNet

Closest papers by meaning

• Evolving Normalization-Activation Layers (NeurIPS 2020 · similarity 0.86)
• CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networks (AAAI 2019 · similarity 0.86)
• Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks (CVPR 2020 · similarity 0.86)
• CIFS: Improving Adversarial Robustness of CNNs via Channel-wise Importance-based Feature Selection (ICML 2021 · similarity 0.86)
• Towards General Robustness Verification of MaxPool-based Convolutional Neural Networks via Tightening Linear Approximation (CVPR 2024 · similarity 0.84)
• Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion (ICLR 2024 · similarity 0.84)