Research map / Architectures and efficiency / RNNs and normalization
From Activation to Initialization: Scaling Insights for Optimizing Neural Fields
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves overparameterization scaling laws for gradient-descent convergence of sine/sinc/Gaussian/wavelet neural fields under standard initializations and proposes a new variance-reduced last-layer initialization that needs fewer parameters.
Problem
Neural fields lack a theoretical account of how many parameters are needed for gradient descent to reach a global minimum as dataset size grows, and how this depends on activation and initialization.
Approach
Shows that the required width scaling depends on activation (periodic/Gaussian-type) and initialization, derives that shrinking the variance of the final layer's weights reduces the overparameterization needed, and proposes two initializations (normal and uniform) based on this, validated on several neural field tasks.
Key results
- Proposed Initialization 1 needs only linear overparameterization in the last hidden layer for shallow nets (vs superlinear for LeCun/Xavier/Kaiming), and quadratic instead of super-quadratic for deep nets.
- On occupancy fitting with Gabor wavelet activations, Initialization 1 reached 22.7 dB train PSNR and 0.89 IoU vs 19.9-21.2 dB and 0.80-0.84 IoU for normal baselines.
- On NeRF Lego with Gaussian activations, Initialization 2 achieved 1.1 dB higher training PSNR than Kaiming uniform and 0.1-1.1 dB higher test PSNR across unseen views.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Initialization 1 instead of standard normal initializations | Gabor wavelet occupancy network · Occupancy Fields | Train PSNR (dB): 20.3 -> 22.7; 21.2 -> 22.7; 19.9 -> 22.7 | Helped |
| Use Initialization 1 instead of standard normal initializations | Gabor wavelet occupancy network · Occupancy Fields | IOU: 0.81 -> 0.89; 0.84 -> 0.89; 0.80 -> 0.89 | Helped |
| Use Initialization 2 instead of standard uniform initializations | Gabor wavelet occupancy network · Occupancy Fields | Train PSNR (dB): 21.2 -> 24.5; 21.6 -> 24.5; 20.2 -> 24.5 | Helped |
| Use Initialization 2 instead of standard uniform initializations | Gabor wavelet occupancy network · Occupancy Fields | IOU: 0.86 -> 0.91; 0.87 -> 0.91; 0.82 -> 0.91 | Helped |
| Use Initialization 2 instead of Kaiming Uniform | Gaussian-activated NeRF · NeRF Lego, novel-view test | PSNR: 0.1 to 1.1dB | Helped |
| Use Initialization 1 for the final layer to reduce required hidden-layer width | Shallow neural fields with sine, sinc, Gaussian, or wavelet activation · Gradient-descent convergence scaling | Required width scaling: n1 = Ω(N 3/2) -> n1 = Ω(N) | Helped |
| Use Initialization 1 for the final layer to reduce required hidden-layer width | Deep neural fields with sine, sinc, or Gaussian activation · Gradient-descent convergence scaling | Required width scaling: nL−1 = Ω(N 5/2) -> nL−1 = Ω(N 2) | Helped |
Surprising findings
- Sinc-activated networks required fewer parameters than the number of training samples beyond about 30,000 samples in the deep image regression experiment, which the authors attribute to the 25 dB cutoff.
Limitations
- Theory applies only to full-batch gradient descent and does not cover mini-batch training.
- Overparameterization result is for convergence to global minimum; very small last-layer variance risks vanishing gradients.
- Experiments use modest-size tasks and PSNR thresholds chosen for memory reasons.
Experimental setup
Datasets: DIV2K NeRF synthetic Lego XYZ RGB Thai statue
Benchmarks: DIV2K 4x super resolution Lego NeRF synthetic Thai statue occupancy Navier-Stokes PINN
Metrics: PSNR SSIM IoU parameter count to convergence
Baselines: LeCun initialization Xavier initialization Kaiming initialization ReLU-PE SIREN uniform initialization
Builds on: Neural Tangent Kernel Nguyen's deep ReLU overparameterization analysis SIREN Fourier feature positional embedding
Closest papers by meaning
- Fast Training of Sinusoidal Neural Fields via Scaling Initialization (ICLR 2025 · similarity 0.88)
- VI^3NR: Variance Informed Initialization for Implicit Neural Representations (CVPR 2025 · similarity 0.87)
- Nonlinear dynamics of localization in neural receptive fields (NeurIPS 2024 · similarity 0.86)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.86)
- How to Train Neural Field Representations: A Comprehensive Study and Benchmark (CVPR 2024 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.