Research map / Architectures and efficiency / RNNs and normalization
Dual-Perspective Activation: Efficient Channel Denoising via Joint Forward-Backward Criterion for Artificial Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual-Perspective Activation (DPA), a parameter-free, end-to-end trainable activation that suppresses irrelevant channels using a per-class joint forward (pre-activation mean) and backward (gradient sign) criterion, evaluated on ViTs, CNNs, GCN/GraphSAGE and BERT/TextGCN.
Problem
Existing activation functions such as ReLU do not fully suppress responses from class-irrelevant channels, and these residual responses act as noise that harms the network's final decision.
Approach
Track per-category moving means of pre-activation global features (forward criterion) and per-category moving averages of binarized post-activation gradient signs (backward criterion); channels failing both criteria are denoised via an added loss penalizing their activation magnitude, with TAU thresholding at zero at inference.
Key results
- On CIFAR-100, DPA raises ViT-Tiny top-1 accuracy to 70.5% versus 65.7% for ReLU and 65.4% for GELU (three-seed averages).
- On ImageNet-1K, DPA reaches 77.8% top-1 with TNT-Small versus 73.4% for ReLU.
- Ablation on CIFAR-100 shows forward-backward intersection (70.5%) outperforms forward-only (68.9%), backward-only (67.9%), union (67.6%), and denoising all channels (65.8%) for ViT-Tiny.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace activations with DPA | ViT-Tiny · CIFAR-10 | Top-1 accuracy (%): 89.9 -> 91.3 | Helped |
| Replace activations with DPA | ViT-Tiny · CIFAR-100 | Top-1 accuracy (%): 65.7 -> 70.5 | Helped |
| Replace activations with DPA | ViT-Tiny · ImageNet-100 | Top-1 accuracy (%): 76.4 -> 80.4 | Helped |
| Replace activations with DPA | ViT-Tiny · ImageNet-1K | Top-1 accuracy (%): 70.9 -> 72.2 | Helped |
| Replace last-block activation with DPA | ResNet-18 · CIFAR-10 | Top-1 accuracy (%): 95.1 -> 95.1 | No clear effect |
| Replace last-block activation with DPA | ResNet-18 · CIFAR-100 | Top-1 accuracy (%): 76.1 -> 76.8 | Helped |
| Replace last-block activation with DPA | ResNet-18 · ImageNet-100 | Top-1 accuracy (%): 84.9 -> 85.7 | Helped |
| Replace last-block activation with DPA | ResNet-18 · ImageNet-1K | Top-1 accuracy (%): 69.7 -> 70.3 | Helped |
| Use DPA instead of ReLU | GCN · DGraph node classification | AUC (%): 72.5 -> 73.7 | Helped |
| Use DPA instead of ReLU | GraphSAGE · DGraph node classification | AUC (%): 75.1 -> 76.6 | Helped |
| Use DPA instead of ReLU | TextGCN · 20News text classification | Top-1 accuracy (%): 86.2 -> 86.9 | Helped |
| Use DPA instead of GELU | BERT · 20News text classification | Top-1 accuracy (%): 86.9 -> 87.8 | Helped |
Surprising findings
- Denoising all channels indiscriminately (DPAall) gives no improvement over ReLU, so channel selection is essential.
- Increasing the activation threshold tau above zero degrades accuracy, so tau=0 is used.
Limitations
- Joint criterion is not always precise and depends on historical statistics during training; the paper admits a remaining gap to the oracle channel-removal setting.
- Category labels are needed to form per-class memories, so the method is tied to classification-style tasks; the authors note it may not apply to box regression in detection.
- Gains of ~1-2 points on some configurations are modest, and reported results are averages over only three seeds without error bars.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1K DGraph 20News
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-100 ImageNet-1K DGraph 20News
Metrics: Top-1 accuracy AUC
Baselines: Softplus ELU SELU SiLU ReLU GELU GDN
Builds on: ReLU Grad-CAM moving mean
Closest papers by meaning
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.84)
- Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected (NeurIPS 2025 · similarity 0.83)
- From Activation to Initialization: Scaling Insights for Optimizing Neural Fields (CVPR 2024 · similarity 0.83)
- Understanding and Improving Optimization in Predictive Coding Networks (AAAI 2024 · similarity 0.83)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.83)
- ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.