Research map / Architectures and efficiency / RNNs and normalization
Fast Training of Sinusoidal Neural Fields via Scaling Initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that simply scaling the initial weights of sinusoidal neural fields (except the last layer) by a constant factor speeds up training up to 10x across image, occupancy, spherical and audio data, with theory on spectral bias and eNTK conditioning.
Problem
Neural fields have high training cost, and the standard SIREN-style initialization for sinusoidal neural fields (SNFs) was designed for signal propagation and final quality rather than training speed.
Approach
Multiply all initialized weights except the last layer by a constant alpha >= 1 ("weight scaling"); this preserves activation distributions (approximately), increases high-frequency content of the initial function and relative power of higher harmonics, and improves the eNTK condition number, accelerating training without hurting generalization at moderate alpha.
Key results
- Weight scaling reaches 42.83 PSNR on KODAK versus 24.58 for SIREN init. after a fixed number of steps, and 45.28 PSNR on ERA5 versus 38.92 for NFSL.
- The paper claims up to 10x training speedup over the standard SNF initialization.
- Weight scaling reduces the eNTK condition number at initialization and during training, while the standard initialization's condition number grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Scale initialized weights in all layers except the last | SNF · Kodak | PSNR: 28.94$\pm$0.21 -> 42.83$\pm$0.35 | Helped |
| Scale initialized weights in all layers except the last | SNF · DIV2K | PSNR: 28.20$\pm$0.13 -> 42.03$\pm$0.41 | Helped |
| Scale initialized weights in all layers except the last | SNF · Lucy | IoU: 0.9936$\pm$0.0002 -> 0.9941$\pm$0.0002 | Helped |
| Scale initialized weights in all layers except the last | SNF · ERA5 | PSNR: 38.92$\pm$0.07 -> 45.28$\pm$0.03 | Helped |
| Scale initialized weights in all layers except the last | SNF · Bach | PSNR: 37.37$\pm$3.11 -> 45.04$\pm$3.23 | Helped |
| Use the higher-frequency initial functional while matching layerwise learning rates to unscaled SNF (ablation) | Weight-scaled SNF · SNF training tasks | training speed | Helped |
| Amplify early-layer learning rates while keeping the initial function at the default (ablation) | SNF · SNF training tasks | training speed | Helped |
| Increase weight scaling to accelerate fitting (ablation) | SNF image regression · unseen coordinates | test PSNR | Hurt |
Surprising findings
- Scaling the initial weights in ReLU networks increases condition number and hurts convergence, the opposite of the effect in sinusoidal networks.
- Most of the speedup comes from the high-frequency initial function rather than from the layerwise learning-rate amplification.
Limitations
- Analysis limited to sinusoidal-activation neural fields rather than general neural field architectures.
- Theory does not give an explicit convergence bound explaining the acceleration.
- The optimal scaling factor is tuned per workload based on structural properties; no automatic selection rule beyond that.
Experimental setup
Datasets: Kodak DIV2K Lucy (Standard 3D Scanning Repository) ERA5 Bach's Cello Suite No. 1
Benchmarks: Kodak DIV2K Lucy ERA5 Bach cello audio
Metrics: PSNR IoU training steps to target PSNR test PSNR
Baselines: Xavier uniform ReLU + P.E. FFN SIREN init. GaussNet MFN WIRE NFSL Sinc NF
Builds on: SIREN lazy training neural tangent kernel Jacobi-Anger identity
Closest papers by meaning
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.90)
- Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks (CVPR 2025 · similarity 0.88)
- From Activation to Initialization: Scaling Insights for Optimizing Neural Fields (CVPR 2024 · similarity 0.88)
- In Search of a Data Transformation That Accelerates Neural Field Training (CVPR 2024 · similarity 0.86)
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.85)
- How to Train Neural Field Representations: A Comprehensive Study and Benchmark (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.