Research map / Architectures and efficiency / RNNs and normalization
Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TUNER, a theory-grounded initialization and weight-bounding scheme for sinusoidal INRs based on an amplitude-phase (Fourier-like) expansion of 3-layer sinusoidal MLPs.
Problem
Initializing and training sinusoidal MLPs (SIREN-style implicit neural representations) is empirical, with unclear frequency generation across layers, leading to noisy reconstructions and overfitting when bandlimit is not controlled.
Approach
An amplitude-phase expansion shows each hidden neuron of a 3-layer sinusoidal MLP is a sum of sines at integer combinations of input frequencies, with amplitudes bounded by a function of hidden weights; this is used to initialize input frequencies as spectral sampling and to clamp/learn hidden-weight bounds that control the bandlimit during training.
Key results
- Frequency-spectrum initialization of input frequencies improves average signal PSNR from 32.2 dB (uniform) to 36.2 dB, a 4 dB gain.
- Bounding hidden weights on a 512x512 image raises signal PSNR from 30.8 to 35.1 dB and gradient PSNR from 15.8 to 27.8 dB.
- TUNER outperforms SIREN and BACON in most (m,n,b) configurations on Kodak signal and gradient PSNR (e.g., b=171, m=n=416: 34.4 vs 29.2 train signal dB for SIREN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform input-frequency initialization with TUNER spectral-sampling initialization | Sinusoidal INR · Kodak images | Signal PSNR (dB): 32.2dB -> 36.2dB; 4dB improvement | Helped |
| Add spectral-sampling initialization with low and high frequency regions; increase high-frequency share from 30% to 70% at l=21 (ablation) | TUNER · Kodak images | Signal PSNR (dB): 35.2 -> 35.0 | Hurt |
| Increase frequency split parameter l from 21 to 64 (ablation) | TUNER · Kodak images | Signal PSNR (dB): 35.2 -> 34.2 at 30%; 35.2 -> 30.8 at 70% | Hurt |
| Add fixed bounds to hidden weights during training (ablation) | TUNER · Kodak test pixels, b=256, cL=1.5, cH=0.05 | Signal PSNR (dB): 32.6 -> 34.8 | Helped |
| Add fixed bounds to hidden weights during training (ablation) | TUNER · Kodak test pixels, b=256, cL=1.5, cH=0.05 | Gradient PSNR (dB): 14.5 -> 25.8 | Helped |
| Increase high-frequency hidden-weight bound cH from 0.05 to 0.2 (ablation) | TUNER with fixed bounds, b=190 and cL=1.5 · Kodak test pixels | Gradient PSNR (dB): 26.1 -> 21.0 | Hurt |
| Add regularization term Lreg to learned bounds (ablation) | TUNER with learned bounds · Kodak image | RGB PSNR (dB): 27.4dB -> 28.5dB | Helped |
| Add regularization term Lreg to learned bounds (ablation) | TUNER with learned bounds · Kodak image | Gradient PSNR (dB): 24.0dB -> 27.1dB | Helped |
| Use TUNER instead of SIREN at b=256 with m=n=416 | Sinusoidal INR · Kodak train pixels | Signal PSNR (dB): 24.7 -> 41.2 | Helped |
| Use TUNER instead of SIREN at b=256 with m=n=416 | Sinusoidal INR · Kodak train pixels | Gradient PSNR (dB): 12.9 -> 23.0 | Helped |
| Use TUNER instead of BACON at b=256 with m=n=416 | Sinusoidal INR · Kodak test pixels | Gradient PSNR (dB): 22.8 -> 21.2 | Hurt |
Surprising findings
- Initializing input frequencies with a larger bandlimit threshold b hurts reconstruction and introduces noise, so smaller b is better even though it limits capacity.
Limitations
- Experiments focus on 2D image fitting with a limited set of images.
- Theory and experiments restricted to 3-layer sinusoidal MLPs; deep networks left for future work.
- Learned bounds require an additional regularization term to prevent overfitting.
Experimental setup
Datasets: Kodak dataset
Benchmarks: Kodak
Metrics: PSNR (signal) PSNR (gradient)
Baselines: SIREN BACON BANF Fourier feature mapping ReLU MLP
Builds on: SIREN Fourier feature mapping BACON Bessel function Fourier series expansion
Closest papers by meaning
- Fast Training of Sinusoidal Neural Fields via Scaling Initialization (ICLR 2025 · similarity 0.88)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.87)
- FINER: Flexible Spectral-bias Tuning in Implicit NEural Representation by Variable-periodic Activation Functions (CVPR 2024 · similarity 0.85)
- A sampling theory perspective on activations for implicit neural representations (ICML 2024 · similarity 0.85)
- FreSh: Frequency Shifting for Accelerated Neural Representation Learning (ICLR 2025 · similarity 0.85)
- Balanced Resonate-and-Fire Neurons (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.