Research map / Probabilistic and scientific modeling / Differential equations and PDEs
ANaGRAM: A Natural Gradient Relative to Adapted Model for efficient PINNs learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ANaGRAM, an empirical natural gradient algorithm with cost O(min(P^2S, S^2P)) that is extended to PINNs and linked to Green's functions, and shows it beats E-NGD, Adam, GD and L-BFGS on four PDE benchmarks.
Problem
Natural gradient methods for training PINNs are computationally prohibitive (cubic in parameter count), and PINN training suffers from ill-conditioning and poor high-precision accuracy.
Approach
Restates natural gradient via an empirical tangent space spanned by batch-restricted NNTK kernels, giving an update computed by SVD of the batch Jacobian at cost linear-ish in parameters; reformulates PINNs as a least-squares regression on the compound model (D,B)∘u, showing the PINN natural gradient update implicitly uses the Green's function of the linearized operator on the tangent space.
Key results
- On the 2D Laplace equation, ANaGRAM reaches a median L2 error of 2.42e-09 versus 1.31e-06 for E-NGD, 1.15e-02 for L-BFGS and 2.05e-03 for Adam.
- On the Allen-Cahn equation, ANaGRAM achieves median L2 error 2.51e-03 versus 3.90e-01 for Adam and 5.40e-01 for L-BFGS, while E-NGD reaches 7.39e-01.
- On the 5D Laplace equation, E-NGD attains the lowest median L2 error (1.40e-05) while ANaGRAM reaches 3.76e-05 with lower wall-clock time than E-NGD (4.88e+02 s vs 4.96e+03 s total).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ANaGRAM instead of E-NGD | PINN training · 2D Laplace equation | L2 error (median): 1.31e-06 -> 2.42e-09 | Helped |
| Use ANaGRAM instead of E-NGD | PINN training · Heat equation | L2 error (median): 2.50e-06 -> 6.48e-07 | Helped |
| Use ANaGRAM instead of E-NGD | PINN training · 5D Laplace equation | L2 error (median): 1.40e-05 -> 3.76e-05 | Hurt |
| Use ANaGRAM instead of E-NGD | PINN training · 5D Laplace equation | test loss (median): 1.51e-08 -> 3.68e-08 | Hurt |
| Use ANaGRAM instead of L-BFGS | PINN training · Allen-Cahn equation | L2 error (median): 5.40e-01 -> 2.51e-03 | Helped |
| Use ANaGRAM instead of Adam | PINN training · Allen-Cahn equation | test loss (median): 7.10e-04 -> 6.22e-05 | Helped |
| Use ANaGRAM instead of Adam | PINN training · 2D Laplace equation | L2 error (median): 2.05e-03 -> 2.42e-09 | Helped |
| Use ANaGRAM instead of Adam | PINN training · Heat equation | test loss (median): 2.37e-04 -> 6.82e-11 | Helped |
Surprising findings
- Adam and gradient descent get stuck at high error levels on most problems, and E-NGD performs poorly on Allen-Cahn (median L2 error 0.739), a nonlinear problem where the claimed equivalence to the natural gradient no longer holds.
- The test loss does not converge for Burgers with viscosity 1e-3, although the training loss does, which the authors attribute to the non-uniform grid emphasising points near the shock.
Experimental setup
Datasets: synthetic PDE collocation points (2D Laplace, 1+1D heat, 5D Laplace, 1+1D Allen-Cahn, Burgers)
Benchmarks: 2D Laplace equation 1+1D heat equation 5D Laplace equation 1+1D Allen-Cahn equation 1+1D Burgers equation (appendix)
Metrics: median L2 error test loss CPU time
Baselines: E-NGD Adam gradient descent with line search L-BFGS
Builds on: physics-informed neural networks natural gradient neural tangent kernel E-NGD
Closest papers by meaning
- Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization (NeurIPS 2025 · similarity 0.88)
- Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks (ICML 2025 · similarity 0.88)
- RoPINN: Region Optimized Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.87)
- Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.87)
- Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective (NeurIPS 2025 · similarity 0.87)
- Sub-Sequential Physics-Informed Learning with State Space Model (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.