Research map / Architectures and efficiency / RNNs and normalization
Sinusoidal Initialization, Time for a New Start
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sinusoidal initialization, a deterministic weight initialization using sine functions with Glorot-scaled amplitude, claiming faster convergence and higher accuracy than default, orthogonal and LSUV initializations across CNNs, ViT and BERT-mini.
Problem
Random initializations (Glorot, He) produce uneven weight distributions and skewed neuron activations, which may slow convergence and reduce final accuracy; deterministic alternatives have not been shown to beat random ones.
Approach
Each weight row is a sinusoid with frequency proportional to row index and phase offset 2πi/m, scaled by an amplitude set to match Glorot variance; rows sum to zero, giving deterministic, centered preactivations and zero skewed neurons at initialization.
Key results
- Sinusoidal initialization improves final validation accuracy by an average 4.9% over default initialization, 1.7% over orthogonal and 3.4% over LSUV.
- AUC (convergence speed) improves on average 20.9% over default, 5.7% over orthogonal and 18.0% over LSUV.
- Skewed neuron proportion under Sinusoidal is 0.2% at both alpha thresholds versus roughly 50-84% for Glorot, He, Orthogonal and LSUV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sinusoidal instead of LSUV initialization | ResNet-50 · CIFAR-100 | maximum validation accuracy (%): 69.7 -> 71.0 | Helped |
| Use Sinusoidal instead of LSUV initialization | ResNet-50 · CIFAR-100 | AUC: 65.5 -> 67.9 | Helped |
| Use Sinusoidal instead of Orthogonal initialization | MobileNetV3 · Tiny-ImageNet | maximum validation accuracy (%): 43.6 -> 42.6 | Hurt |
| Use Sinusoidal instead of Orthogonal initialization | MobileNetV3 · Tiny-ImageNet | AUC: 84.7 -> 82.0 | Hurt |
| Use Sinusoidal instead of Orthogonal initialization | EfficientNetV2 · Tiny-ImageNet | maximum validation accuracy (%): 50.2 -> 53.5 | Helped |
| Use Sinusoidal instead of Orthogonal initialization | EfficientNetV2 · Tiny-ImageNet | AUC: 100.3 -> 106.4 | Helped |
| Use Sinusoidal instead of LSUV initialization | ViT-B16 · ImageNet-1k | maximum validation accuracy (%): 29.6 -> 31.5 | Helped |
| Use Sinusoidal instead of Orthogonal initialization | BERT-mini · WikiText | maximum validation accuracy (%): 42.2 -> 41.1 | Hurt |
| Use Sinusoidal instead of Orthogonal initialization | BERT-mini · WikiText | AUC: 71.6 -> 72.0 | Helped |
| Replace MobileNetV3's HardSwish activation with ReLU while using Sinusoidal initialization (ablation) | MobileNetV3 · Tiny-ImageNet | validation accuracy | Helped |
| Use Sinusoidal instead of LSUV initialization | ViT-B16 · ImageNet-1k | skewed neurons (%) at alpha 0.1: 84.3 -> 0.2 | Helped |
| Use Sinusoidal instead of He initialization | ViT-B16 · ImageNet-1k | OUI: 0.59 -> 0.98 | Helped |
Surprising findings
- Orthogonal initialization outperforms Sinusoidal on final accuracy for MobileNetV3 on Tiny-ImageNet with SGD and Adam settings and on the MobileNetV3 AUC.
- Sinusoidal initialization underperforms on EfficientNetV2 with SGD at 10 epochs (9.0% vs LSUV 13.4%).
Limitations
- Results are not consistent across all settings; MobileNetV3 falls behind orthogonal initialization.
- Theoretical guarantees of no skewed neurons are proven only for MLPs; extension to other architectures is open.
- No learning rate schedulers used, so accuracies may not match SOTA benchmarks.
Experimental setup
Datasets: CIFAR-100 Tiny-ImageNet ImageNet-1k WikiText-2
Benchmarks: CIFAR-100 Tiny-ImageNet ImageNet-1k WikiText-2
Metrics: validation accuracy area under the training curve (AUC) skewed neuron proportion orthogonality-based unique input index (OUI)
Baselines: Default initialization Orthogonal initialization LSUV Glorot initialization He initialization
Builds on: Glorot initialization He initialization Orthogonal initialization LSUV sinusoidal positional encoding
Closest papers by meaning
- Fast Training of Sinusoidal Neural Fields via Scaling Initialization (ICLR 2025 · similarity 0.90)
- IDInit: A Universal and Stable Initialization Method for Neural Network Training (ICLR 2025 · similarity 0.87)
- Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks (CVPR 2025 · similarity 0.87)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- From Activation to Initialization: Scaling Insights for Optimizing Neural Fields (CVPR 2024 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.