Research map / Probabilistic and scientific modeling / Differential equations and PDEs
An operator preconditioning perspective on training in physics-informed machine learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that gradient descent training of physics-informed ML models (linearized/NTK regime) converges at a rate governed by the condition number of an operator composing the Hermitian square of the PDE operator with the tangent-kernel integral operator, and proposes parameter rescaling (preconditioning) of Fourier features to fix ill-conditioning on Poisson, Helmholtz and advection equations.
Problem
Physics-informed machine learning (PINNs and variants) trains slowly or fails with gradient descent, and there is little principled analysis of why training is hard.
Approach
Under approximate linearity (small Hessian/NTK drift), GD on the physics-informed loss reduces to a simplified linear iteration whose convergence rate is set by the condition number of matrix A = D*D-based Gram-type operator, which equals the condition number of the Hermitian square D*D composed with the tangent kernel integral operator; preconditioning by rescaling parameters (e.g. Fourier featur
Key results
- For the 1-D Poisson equation with Fourier features, the unpreconditioned matrix condition number grows at least as K^4, while a rescaled (preconditioned) version has condition number bounded by a constant independent of K, with loss decaying exponentially fast.
- For linear advection, the unpreconditioned condition number scales as beta^2 while the preconditioned one stays constant, and the preconditioned Fourier model trains fast and outperforms an MLP.
- Theorem shows kappa(A) >= kappa(A_op∘TT*) with equality when the Gram matrix is invertible, linking training speed to the conditioning of the Hermitian square operator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Rescale Fourier features using the differential-operator spectrum | Fourier features for the Poisson equation · Poisson equation | training loss decay | Helped |
| Rescale Fourier features using the differential-operator spectrum | Fourier features for the Poisson equation · Poisson equation | condition number: \kappa(\sA(\lambda)) \geq K^4; \kappa(\Tilde{\sA}(2\pi/\gamma^2, \gamma)) = 1 + O(1/\gamma) | Helped |
| Precondition Fourier features according to the PDE operator | Fourier features for the linear advection equation · Linear advection equation | training speed | Helped |
| Increase the advection speed β | Unpreconditioned Fourier features · Linear advection equation | condition number: \sA(\beta) \sim \beta^2 | Hurt |
| Impose hard boundary conditions by multiplying the model by sin(x) | Poisson equation toy model · Poisson equation | condition number: 3+2\sqrt{2}\approx 5.83 -> 4 | Helped |
| Impose hard boundary conditions by subtracting uθ(π) | Poisson equation toy model · Poisson equation | condition number: 3+2\sqrt{2}\approx 5.83 -> 1 | Helped |
| Choose λ to minimize the condition number | Fourier features for the Poisson equation · Poisson equation | condition number | Helped |
| Use hard boundary conditions instead of soft boundary conditions | Fourier features for the linear advection equation · Linear advection equation | condition number | Helped |
Surprising findings
- Optimal lambda for the boundary-loss weight scales as K^2 for Poisson, whereas existing adaptive weighting schemes (learning rate annealing, NTK-based) yield scalings like K^3.5 and K^4, which are suboptimal for conditioning.
Limitations
- Analysis and experiments focus on linear PDEs and linear-in-parameter models; extension to nonlinear PDEs is future work.
- Does not provide a universal preconditioning strategy for nonlinear models such as neural networks.
- Results rely on the NTK/approximate-linearity regime, excluding feature learning.
Experimental setup
Metrics: condition number training loss
Baselines: unpreconditioned Fourier features MLP learning rate annealing (soft BC weighting) NTK-based lambda weighting
Builds on: PINNs neural tangent kernel learning rate annealing for PINNs causal training for PINNs
Closest papers by meaning
- Learning a Neural Solver for Parametric PDEs to Enhance Physics-Informed Methods (ICLR 2025 · similarity 0.90)
- Dual Cone Gradient Descent for Training Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.88)
- Improving Energy Natural Gradient Descent through Woodbury, Momentum, and Randomization (NeurIPS 2025 · similarity 0.88)
- Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.88)
- Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning (NeurIPS 2024 · similarity 0.88)
- The Challenges of the Nonlinear Regime for Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.