Research map / 3D vision / NeRF and radiance fields

Preconditioners for the Stochastic Training of Neural Fields

CVPR 2025 · 3D vision › NeRF and radiance fields · approach: NeRF for scene decomposition and understanding · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a theoretical framework showing that curvature-aware diagonal preconditioners (ESGD and others) accelerate stochastic training of neural fields with sine, Gaussian, or wavelet activations but not ReLU(PE), and validates this on image reconstruction, occupancy, and NeRF.

Problem

Adam is the de facto optimizer for stochastic training of neural fields but requires long training times; second-order methods like L-BFGS fail in stochastic settings, and it is unclear when preconditioning helps.

Approach

Views Adam as applying a diagonal Gauss-Newton preconditioner and shows via Hessian-vector sparsity theorems that sine/Gaussian/wavelet/sinc activated neural fields have dense Hessian-vector products (so curvature-aware diagonal preconditioners like ESGD help), whereas ReLU/ReLU(PE) networks have sparse ones (so preconditioning gives little benefit).

Key results

• In 2D image reconstruction, a wavelet neural field trained with ESGD reaches 29.36 PSNR at epoch 10 versus 26.61 for Adam.
• On NeRF LLFF scenes, ESGD reaches the target training PSNR in fewer iterations on average than Adam (e.g., flower 120K vs 200K iterations), with comparable test PSNR.
• ESGD achieves faster convergence than AdaHessian, Shampoo, and Kronecker-factored preconditioners on Gaussian image reconstruction and occupancy tasks, with per-iteration cost close to Adam.

Measured effects reported in the paper

Surprising findings

• Adam outperforms ESGD for ReLU(PE) neural fields, consistent with the theory that their Hessian-vector products are sparse.
• Shampoo and AdaHessian lose their advantage on the higher-frequency 3D occupancy task, unlike in 2D image reconstruction.

Limitations

• Ineffective for ReLU(PE) neural fields, which Adam handles better; no alternative preconditioner explored for them.
• Results are mainly for Gaussian activations in the main text; other activations and tasks are relegated to supplement.
• Theory is restricted to sparsity/density of Hessian-vector products rather than a full convergence analysis.

Experimental setup

Datasets: DIV2K Stanford 3D Scanning (armadillo, dragon) LLFF Blender synthetic BLEFF

Benchmarks: DIV2K Stanford 3D Scanning LLFF

Metrics: MSE loss PSNR IoU SSIM LPIPS training time per iteration memory per iteration

Baselines: Adam SGD AdaHessian Shampoo K-FAC L-BFGS

Builds on: ESGD AdaHessian K-FAC Shampoo Adam

Closest papers by meaning

• On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.87)
• On Convergence of Adam for Stochastic Optimization under Relaxed Assumptions (NeurIPS 2024 · similarity 0.86)
• Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.85)
• AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.85)
• Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.85)
• No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.85)