Research map / Optimization / Optimizers

Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks

ICML 2025 · Optimization › Optimizers · approach: Natural gradient and Fisher approximations

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes sketchy natural gradient descent (SNGD), a randomized single-/two-pass sketching method that approximates the natural energy gradient for PINN training, exploiting rapidly decaying Gram matrix eigenvalues.

Problem

Natural energy gradient methods for PINNs achieve high accuracy but require forming and inverting the dense |θ|×|θ| Gram matrix, which is prohibitively costly and memory-intensive for all but small networks.

Approach

Replace the full Gram matrix with a randomized low-rank eigendecomposition obtained via Gaussian sketching (single-pass or two-pass RSVD), justified by the exponentially decaying Gram eigenvalues so the pseudoinverse approximation error is near machine precision.

Key results

• On the heat equation with a 5,301-parameter network, two-pass/single-pass SNGD reaches median relative L2 error ~1e-8 vs 5.66e-7 for ENGD, and trains in ~1.5–1.9 min vs 221.9 min for ENGD.
• SNGD trains a PINN with over 1 million parameters (heat equation) within minutes to error ~1.5e-8, where full ENGD and BFGS are infeasible in memory.
• On the transport equation with c=30, SNGD achieves relative L2 errors of about 2.7e-4 to 8e-4 (two-pass) across five runs, while ENGD fails to converge in three of five runs.

Measured effects reported in the paper

Surprising findings

• Sketched natural gradient often attains lower error than the exact natural gradient (ENGD), which the authors attribute to randomness and row-mixing from distinct Gaussian sketches each iteration finding flatter minima.
• Gram matrix rank at the end of training is larger under SNGD than ENGD, despite using a low-rank approximation.

Limitations

• Algorithm is sensitive to the truncation tolerance hyperparameter.
• The rapid eigenvalue decay is proven only for a one-layer, one-dimensional ReLU network; for multi-layer settings it is empirical.
• Experiments are limited to small-to-moderate PDE problems on a single A100 GPU.

Experimental setup

Datasets: collocation points sampled on PDE domains (heat, Poisson, nonlinear BVP, transport equation)

Benchmarks: 1D heat equation 2D Poisson equation nonlinear boundary value problem 1D transport equation with c=30

Metrics: relative L2 error loss wall-clock runtime (minutes) Gram matrix truncated rank

Baselines: ENGD ADAM BFGS

Builds on: energy natural gradient descent (ENGD) randomized SVD (Halko et al. 2011) single-pass and two-pass sketching (Saibaba et al. 2016)

Closest papers by meaning

• Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization (NeurIPS 2025 · similarity 0.92)
• Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.91)
• ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning (ICLR 2025 · similarity 0.88)
• Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks (NeurIPS 2026 · similarity 0.88)
• ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks (ICLR 2025 · similarity 0.88)
• Dual Cone Gradient Descent for Training Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.87)