Research map / Probabilistic and scientific modeling / Differential equations and PDEs
TENG: Time-Evolving Natural Gradient for Solving PDEs With Deep Neural Nets Toward Machine Precision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TENG, a sequential-in-time neural PDE solver that repeatedly projects function-space gradient steps onto the network parameter space (natural gradient / least-squares), with Euler and Heun integrators, reaching near machine-precision per-step loss on heat, Allen-Cahn and Burgers' equations.
Problem
Neural-network PDE solvers for initial value problems lose accuracy because errors accumulate over time; existing sequential-in-time methods (TDVP, OBTI) are either projection-limited or hard to optimize.
Approach
Generalize TDVP and OBTI by performing multiple iterations per physical time step, each one projecting a function-space gradient onto parameter space via a randomly sub-sampled least-squares (natural gradient / Gauss-Newton) solve, plus a Heun-type two-stage integrator that avoids high-order differentiation of the PDE operator.
Key results
- TENG-Heun achieves global relative L2 error of 1.588e-6 on 2D heat, 1.139e-5 on 3D heat, 6.187e-6 on Allen-Cahn and 2.643e-6 on Burgers', versus 1.15e-5 for the best baseline (PINN-BFGS) on 2D heat.
- TENG-Euler per-step training loss reaches near machine precision (~1e-14) within about 7 iterations, while OBTI reaches only ~1e-7 after hundreds of iterations.
- TENG-Euler is comparable to or better than TDVP-RK4 in global error across all four benchmarks (e.g., 3.006e-4 vs 3.279e-4 on 2D heat).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Heun instead of Euler integration in TENG (ablation) | TENG · Heat equation 2D | global relative L2-error: 3.006\times10^{-4} -> 1.588\times10^{-6} | Helped |
| Use Heun instead of Euler integration in TENG (ablation) | TENG · Heat equation 3D | global relative L2-error: 3.664\times10^{-4} -> 1.139\times10^{-5} | Helped |
| Use Heun instead of Euler integration in TENG (ablation) | TENG · Allen-Cahn | global relative L2-error: 1.249\times10^{-3} -> 6.187\times10^{-6} | Helped |
| Use Heun instead of Euler integration in TENG (ablation) | TENG · Burgers | global relative L2-error: 3.598\times10^{-4} -> 2.643\times10^{-6} | Helped |
| Use TENG-Heun instead of PINN-BFGS | PDE solution methods · Heat equation 2D | global relative L2-error: 1.150\times10^{-5} -> 1.588\times10^{-6} | Helped |
| Use TENG-Heun instead of TDVP-RK4 | PDE solution methods · Heat equation 3D | global relative L2-error: 3.841\times10^{-3} -> 1.139\times10^{-5} | Helped |
| Use TENG-Heun instead of TDVP-RK4 | PDE solution methods · Allen-Cahn | global relative L2-error: 1.258\times10^{-3} -> 6.187\times10^{-6} | Helped |
| Use TENG-Heun instead of TDVP-RK4 | PDE solution methods · Burgers | global relative L2-error: 2.437×10^{-3} -> 2.643\times10^{-6} | Helped |
Surprising findings
- TENG-Heun with the smallest time step is not always more accurate than with larger steps, because per-step accumulation dominates at small dt while discretization error dominates at large dt.
Limitations
- Experiments limited to 2D and 3D scalar PDEs with periodic boundary conditions (plus one Dirichlet disk case for heat).
- Global convergence of the natural gradient is not guaranteed in general; the paper relies on convexity in function space and small time steps.
- Error from forward Euler discretization dominates TENG-Euler; higher-order schemes needed for best accuracy.
Experimental setup
Datasets: analytical heat equation solutions spectral method solutions for Allen-Cahn and Burgers' equations
Benchmarks: 2D heat equation 3D heat equation Allen-Cahn equation Burgers' equation
Metrics: global relative L2 error relative L2 error per time step training loss (squared L2) runtime
Baselines: TDVP-RK4 OBTI-Adam OBTI-LBFGS PINN-ENGD PINN-BFGS
Builds on: time-dependent variational principle optimization-based time integration PINN energy natural gradient (ENGD)
Closest papers by meaning
- Interpretable Solutions for Multi-Physics PDEs Using T-NNGP (AAAI 2025 · similarity 0.87)
- Learning a Neural Solver for Parametric PDEs to Enhance Physics-Informed Methods (ICLR 2025 · similarity 0.86)
- Number Theoretic Accelerated Learning of Physics-Informed Neural Networks (AAAI 2025 · similarity 0.86)
- UGrid: An Efficient-And-Rigorous Neural Multigrid Solver for Linear PDEs (ICML 2024 · similarity 0.86)
- MultiPDENet: PDE-embedded Learning with Multi-time-stepping for Accelerated Flow Simulation (ICML 2025 · similarity 0.86)
- Dual Cone Gradient Descent for Training Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.