Research map / Learning theory and algorithms / Kernels and regression theory
The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that for nonlinear PDEs the PINN NTK is random at initialization and non-constant during training, with a non-vanishing Hessian even in the infinite-width limit, and argues for second-order (Levenberg-Marquardt) optimization.
Problem
NTK theory explains PINN training dynamics for linear PDEs (deterministic, constant NTK), but it is unclear whether it applies to nonlinear PDEs in the infinite-width limit.
Approach
Proves that with NTK-scaling (1/sqrt(m)) one-hidden-layer networks, the NTK for nonlinear PDE residuals factorizes into a deterministic linear part and a random nonlinear part depending on the gradient of R, so it is random at init and changes during training; the residual Hessian stays bounded away from zero, motivating Levenberg-Marquardt training.
Key results
- For Burgers' equation, PINNs trained with Levenberg-Marquardt reach relative L2 precision around 1e-5 in few iterations, versus about 1e-3 for Adam and L-BFGS.
- On the convection equation with beta=100, LM without other enhancements achieves high accuracy, which the paper says Adam with curriculum training cannot reach (manages beta only up to 20).
- Theory: the NTK at initialization converges in distribution to a random limit for nonlinear PDEs, and the residual Hessian norm is lower-bounded by a constant as width grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Levenberg-Marquardt instead of Adam | Vanilla PINN · Burgers' equation | relative L2 loss: approximately $10^{-3}$ -> around $10^{-5}$ | Helped |
| Use Levenberg-Marquardt instead of L-BFGS | Vanilla PINN · Burgers' equation | relative L2 loss: approximately $10^{-3}$ -> around $10^{-5}$ | Helped |
| Use Levenberg-Marquardt instead of Adam with random Fourier features | PINN · Poisson equation | relative L2 loss | Helped |
| Add random Fourier features | Levenberg-Marquardt PINN · Poisson equation | relative L2 loss | Helped |
| Use Levenberg-Marquardt instead of Adam | PINN · Convection equation | L2 loss | Helped |
| Add curriculum training | Levenberg-Marquardt PINN · Convection equation | L2 loss | Helped |
| Use Levenberg-Marquardt without other enhancements instead of Adam with curriculum training | PINN · Convection equation, beta = 100 | L2 loss | Helped |
| Use Levenberg-Marquardt instead of Adam | PINN with causality training · Navier-Stokes equation | relative L2 loss | Helped |
Surprising findings
- Even in the infinite-width limit the NTK of nonlinear PDE PINNs is random and changes during training, contrary to the conventional NTK picture.
- Quasi-Newton methods such as L-BFGS fall in between first- and second-order methods in performance on Burgers' equation.
Limitations
- Second-order methods scale poorly with the number of parameters; storing J^T J is infeasible for medium-to-large networks.
- Theory is for one-hidden-layer networks; extension to deep networks is left open.
- Full-rankness of the NTK in the nonlinear case and convergence to global minima are not established.
Experimental setup
Datasets: Burgers' equation Poisson equation convection equation wave equation Navier-Stokes 2D cylinder wake
Benchmarks: Burgers' equation Poisson equation convection equation Navier-Stokes 2D cylinder wake
Metrics: relative L2 loss spectral norm of NTK relative NTK change Hessian spectral norm
Baselines: Adam L-BFGS loss balancing random Fourier features curriculum training causality training
Builds on: neural tangent kernel physics-informed neural networks Adam L-BFGS
Closest papers by meaning
- Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective (NeurIPS 2025 · similarity 0.89)
- PRDP: Progressively Refined Differentiable Physics (ICLR 2025 · similarity 0.88)
- Efficient Error Certification for Physics-Informed Neural Networks (ICML 2024 · similarity 0.88)
- An operator preconditioning perspective on training in physics-informed machine learning (ICLR 2024 · similarity 0.87)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.87)
- Riemann Tensor Neural Networks: Learning Conservative Systems with Physics-Constrained Networks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.