Research map / Architectures and efficiency / RNNs and normalization
Sparser, Better, Deeper, Stronger: Improving Static Sparse Training with Exact Orthogonal Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Exact Orthogonal Initialization (EOI), which samples sparse orthogonal weight matrices by composing random Givens rotations until a target density is reached, and uses them as static sparse initializations for MLPs and CNNs.
Problem
Static sparse training initializes masks from dense initializations, ignoring how pruning destroys signal propagation; existing orthogonal sparse initializations are only approximate or restricted in architecture and sparsity levels.
Approach
Sample the sparse mask and weights jointly by multiplying random Givens rotations (each O(n)), giving exactly orthogonal sparse matrices at arbitrary density; for convolutions, embed the orthogonal matrix in kernel centers (delta-orthogonal) and spread the mask over the kernel.
Key results
- On 1000-layer CNN (CIFAR-10, density 12.5%), ERK-EOI reaches 78.41% vs 39.08% for ERK-AI and 10.00% for ERK-Base.
- On ResNet32 CIFAR-10 at 10% density, ERK-EOI gets 89.79% vs 89.49% for ERK-Base.
- On DeiT III ImageNet at density 0.1, ERK-EOI gives about 1% gain over ERK-Base; on ResNet50 the gain is not significant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EOI instead of AI with ERK density distribution | ResNet32 · CIFAR-10, density 10% | test accuracy: 89.51(0.21) -> 89.79(0.38) | Helped |
| use EOI instead of ERK Base initialization | ResNet56 · CIFAR-10, density 10% | test accuracy: 90.82(0.23) -> 91.07(0.15) | Helped |
| use EOI instead of AI with ERK density distribution | ResNet110 · CIFAR-10, density 10% | test accuracy: 91.93(0.20) -> 91.66(0.35) | Hurt |
| use EOI instead of AI with ERK density distribution | VGG-16 · CIFAR-10, density 10% | test accuracy: 91.51(0.10) -> 92.02(0.12) | Helped |
| use EOI instead of ERK Base initialization | EfficientNet · Tiny ImageNet, density 10% | test accuracy: 45.54(0.76) -> 44.13(0.77) | Hurt |
| use EOI instead of AI with ERK density distribution | ResNet110 · CIFAR-100, density 10% | test accuracy: 66.97(0.10) -> 66.97(0.34) | No clear effect |
| use EOI instead of SAO | 1000-layer MLP · MNIST, density 12.5% | test accuracy: 96.74(0.13) -> 95.98(0.25) | Hurt |
| use EOI instead of SAO | 1000-layer CNN · CIFAR-10, density 12.5% | test accuracy: 78.80(0.11) -> 78.41(0.15) | Hurt |
| replace uniform random Givens angles with fixed φ = π / 180 (ablation) | EOI on ResNet32 · CIFAR-10, density 10% | initial training loss | Hurt |
| replace uniform random Givens angles with fixed φ = π / 180 (ablation) | EOI on ResNet32 · CIFAR-10, density 10%, after 200 epochs | test accuracy: 89.58(0.09) -> 89.65(0.11) | No clear effect |
| replace uniform random Givens angles with fixed φ = π · 80 / 180 (ablation) | EOI on ResNet32 · CIFAR-10, density 10%, after 200 epochs | test accuracy: 89.58(0.09) -> 89.87(0.22) | No clear effect |
| increase the density of convolutional center matrices H from d^l to √d^l (ablation) | EOI on ResNets and VGG · CIFAR-10 and CIFAR-100, density 10% | test accuracy | Helped |
Surprising findings
- Plain sparse masks from density-distribution methods (Base) with 1000-layer networks give random performance (about 11% on MNIST, 10% on CIFAR-10).
- Orthogonal sparse initialization improves training even with ReLU activations in most cases, despite dynamical isometry being unattainable there.
Limitations
- Evaluation limited to image recognition tasks; no NLP or large language model experiments.
- AI approach and EOI gaps are small in architectures with residual connections and normalization, and EOI underperforms on EfficientNet with bottleneck blocks.
- Convolution orthogonality is only approximated by delta-orthogonal embedding and mask spreading, and density of central matrix requires tuning (sqrt(d) choice).
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet
Benchmarks: MNIST CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet
Metrics: test accuracy mean and max singular values of input-output Jacobian orthogonality score wall-time
Baselines: Uniform ERK SNIP GraSP Synflow Approximated Isometry SAO
Builds on: delta-orthogonal initialization approximated isometry SAO SNIP GraSP Synflow ERK
Closest papers by meaning
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.87)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.87)
- Sparse-IFT: Sparse Iso-FLOP Transformations for Maximizing Training Efficiency (ICML 2024 · similarity 0.86)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.86)
- Dynamic Sparse Training with Structured Sparsity (ICLR 2024 · similarity 0.86)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.