Research map / Architectures and efficiency / RNNs and normalization
DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that shifted-ReLU and soft-threshold sparsifying activations are untrainable at the edge of chaos in deep nets due to an unstable variance-map fixed point, and proposes magnitude-clipped variants (CReLU, CST) that restore stability and train with up to 85% activation sparsity.
Problem
Sparsifying activations (shifted ReLU and soft thresholding) that induce high fractional sparsity at initialization fail to train in very deep DNNs and CNNs, because at the edge of chaos their Gaussian process variance map has an unstable fixed point.
Approach
Clip the activation magnitude at level m (CReLU_{tau,m}, CST_{tau,m}); this makes V'(q*)<1 while keeping chi_1=1 at the EoC, so the variance fixed point is locally stable, enabling trainable deep sparse networks at up to 85% sparsity.
Key results
- CReLU and CST DNNs (100 layers, MNIST) retain near full accuracy (~94-95%) with activation sparsity up to 85% when trained with SGD.
- Unclipped ReLU_tau and ST_tau networks fail to train consistently at 60% and 50% sparsity respectively.
- CNNs (50 layers, CIFAR10) with clipped activations retain full accuracy up to 70% sparsity and reach ~66% accuracy at 85% sparsity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add magnitude clipping to shifted ReLU | 100-layer DNN · MNIST | test accuracy: 0.10 -> 0.92 | Helped |
| add magnitude clipping to SoftThreshold | 100-layer DNN · MNIST | test accuracy: 0.10 -> 0.92 | Helped |
| use shifted ReLU at s=0.60 instead of standard ReLU | 100-layer DNN · MNIST | test accuracy: 0.94 -> 0.76 | Hurt |
| increase CReLU clipping magnitude from m=0.81 to m=1.17 (ablation) | 100-layer DNN · MNIST, s=0.85 | test accuracy: 0.78 -> 0.94 | Helped |
| increase CReLU clipping magnitude from m=1.17 to m=1.74 (ablation) | 100-layer DNN · MNIST, s=0.85 | test accuracy: 0.94 -> 0.28 | Hurt |
| increase CST clipping magnitude from m=0.67 to m=1.00 (ablation) | 100-layer DNN · MNIST, s=0.85 | test accuracy: 0.76 -> 0.93 | Helped |
| increase CST clipping magnitude from m=1.00 to m=1.53 (ablation) | 100-layer DNN · MNIST, s=0.85 | test accuracy: 0.93 -> 0.14 | Hurt |
| increase CST clipping magnitude from m=0.72 to m=1.06 (ablation) | 100-layer DNN · MNIST, s=0.80 | test accuracy: 0.92 -> 0.95 | Helped |
| increase CST clipping magnitude from m=1.06 to m=1.61 (ablation) | 100-layer DNN · MNIST, s=0.80 | test accuracy: 0.95 -> 0.11 | Hurt |
| use clipped activations at s=0.70 rather than unclipped thresholded activations | 50-layer CNN · CIFAR10 | test accuracy: 0.1 -> 0.70 | Helped |
| use clipped activations at s=0.85 | 50-layer CNN · CIFAR10 | test accuracy: 0.63 -> 0.66 | Mixed |
Surprising findings
- Standard EoC-initialized shifted ReLU and soft-threshold networks are unstable because V''(q*)>0 for any tau>0, so they fail to train even at moderate sparsity.
- Maximum accuracy for DNNs increases with sparsity up to a point (e.g. ReLU at 80-85% sparsity reaches 94%).
Limitations
- Experiments are limited to 100-layer MNIST DNNs and 50-layer CIFAR10 CNNs with a single run per CNN setting.
- Large m can cause multiple fixed points and exploding gradients; the stable range of m is only heuristically chosen.
- No analysis of transformers or residual networks.
Experimental setup
Datasets: MNIST CIFAR10
Benchmarks: MNIST CIFAR10
Metrics: test accuracy test sparsity V'(q*) V''(q*)
Baselines: ReLU ReLU_tau ST_tau CReLU_{tau,m} CST_{tau,m}
Builds on: edge of chaos initialization dynamical isometry FATReLU
Closest papers by meaning
- Exploring the Benefit of Activation Sparsity in Pre-training (ICML 2024 · similarity 0.88)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.88)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.87)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.87)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.87)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.