Research map / 3D vision / Face and shape
VI^3NR: Variance Informed Initialization for Implicit Neural Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a variance-stable weight initialization for arbitrary activations in implicit neural representations (INRs), using Monte Carlo or analytic activation statistics and a forward/backward variance condition, and shows gains for Gaussian and sinc INRs on image, SDF and audio reconstruction.
Problem
Standard neural network initializations (Xavier, Kaiming, PyTorch gains) rely on assumptions specific to tanh/sigmoid/ReLU and do not apply well to INR activations such as Gaussian, sinc, and sine, leading to poor convergence and accuracy in signal reconstruction.
Approach
Derive weight variance from the CLT so pre-activations have a chosen variance sigma_p^2 at each layer, estimate activation statistics via Monte Carlo sampling, and pick sigma_p by equating forward and backward variance conditions, which generalizes Xavier, Kaiming, SIREN and PyTorch gain initializations.
Key results
- On KODIM images, Gaussian INR with the analytic init reaches 79.73 PSNR versus 30.42 for Gaussian with random normal init and 55.77 for SIREN init.
- On BACON SDF reconstruction, Gaussian with the analytic init achieves 0.9697 IoU versus 0.8628 for random normal init, though SIREN remains higher (0.9791).
- Audio MSE for Gaussian with the analytic init drops to 0.48e-3, 1.49e-3, 0.24e-3 across the three signals, versus 6.57e-3, 5.10e-3, 1.35e-3 with random normal init.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed initialization instead of random normal initialization | Sinc INR · KODIM | PSNR: 32.11 -> 71.44 | Helped |
| Use the proposed initialization instead of random normal initialization | Gaussian INR · KODIM | PSNR: 30.42 -> 79.73 | Helped |
| Use the proposed SIREN initialization instead of random normal initialization | SIREN · KODIM | PSNR: 10.71 -> 55.77 | Helped |
| Use the proposed initialization instead of random normal initialization | Sinc INR · BACON shapes | IoU: 0.7252 -> 0.9055 | Helped |
| Use the proposed initialization instead of random normal initialization | Gaussian INR · BACON shapes | IoU: 0.8628 -> 0.9697 | Helped |
| Use the proposed initialization instead of random normal initialization | Gaussian INR · Bach audio segment | MSE (×10−3): 6.57 -> 0.48 | Helped |
| Use the proposed initialization instead of random normal initialization | Gaussian INR · Counting audio segment | MSE (×10−3): 5.10 -> 1.36 | Helped |
| Use the proposed initialization instead of random normal initialization | Gaussian INR · Two Speakers audio segment | MSE (×10−3): 1.35 -> 0.24 | Helped |
| Replace analytical activation statistics with Monte Carlo estimates (ablation) | Gaussian INR · KODIM | PSNR: 79.73 -> 75.85 | Hurt |
| Replace analytical activation statistics with Monte Carlo estimates (ablation) | Gaussian INR · BACON shapes | IoU: 0.9697 -> 0.9693 | No clear effect |
| Use the proposed Gaussian initialization instead of random normal initialization (ablation) | Gaussian INR · Two Speakers audio segment | MSE (×10−3): 1.35 -> 0.24 | Helped |
| Set σp using the theoretical backward condition instead of task grid search (ablation) | Gaussian INR · KODIM | PSNR: 79.73 -> 25.08 | Hurt |
Surprising findings
- Sine-activation SIREN initialization is already optimal for sigma_p=1, yet SIREN still remains slightly ahead of the proposed Gaussian/sinc init on 3D SDF reconstruction.
- The optimal preactivation scale appears to depend on the task data distribution rather than only on input/output dimensions.
Limitations
- Backward-pass condition matters mainly for very deep networks; the experiments use shallow 8-layer, 128/256-unit networks.
- The best sigma_p is task-specific and still requires a local line search beyond the theoretical value.
- The proposed init does not beat SIREN on 3D SDF reconstruction.
Experimental setup
Datasets: KODAK (KODIM) image dataset BACON 3D shapes SIREN audio signals (Bach, Counting, Two Speakers)
Benchmarks: KODIM image reconstruction (PSNR) BACON 3D SDF reconstruction (IoU) SIREN audio reconstruction (MSE)
Metrics: PSNR IoU MSE SMAPE forward/backward error
Baselines: SIREN init random normal init Xavier init Kaiming init Kumar init PyTorch gain init
Builds on: SIREN Xavier initialization Kaiming initialization Kumar initialization Gaussian activation INR
Closest papers by meaning
- Implicit Representations via Operator Learning (ICML 2024 · similarity 0.90)
- MIRE: Matched Implicit Neural Representations (CVPR 2025 · similarity 0.89)
- Few-shot Implicit Function Generation via Equivariance (CVPR 2025 · similarity 0.89)
- Grids Often Outperform Implicit Neural Representation at Compressing Dense Signals (NeurIPS 2025 · similarity 0.88)
- Learning Transferable Features for Implicit Neural Representations (NeurIPS 2024 · similarity 0.88)
- Bias for Action: Video Implicit Neural Representations with Bias Modulation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.