Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a gradient alignment score to diagnose directional gradient conflicts in PINN training and shows that quasi-second-order optimizers, especially SOAP, improve alignment and accuracy over Adam on 10 PDE benchmarks.
Problem
PINN training suffers from directional gradient conflicts between loss terms (initial/boundary and residual), slowing convergence and limiting accuracy, especially for complex and turbulent flows.
Approach
Defines a multi-vector alignment score (generalizing cosine similarity) measuring intra-step and inter-step gradient conflicts, theoretically links quasi-second-order preconditioning to higher alignment, and argues SOAP approximates a Newton preconditioner.
Key results
- SOAP reduces relative L2 error on the Kolmogorov flow at Re=10^4 to 3.20e-2 versus 2.04e-1 for Adam (the baseline pipeline's 20.4%).
- On the lid-driven cavity at Re=5000, SOAP reduces relative L2 error from 32.4% to 3.99%.
- SOAP achieves the best error on 8 of 10 PDE benchmarks reported, with 6.9x improvement on Allen-Cahn over the baseline and about 2x on KdV and KS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace Adam with SOAP | PirateNet PINN · Wave | relative L^2 error: 5.15 \times 10^{-5} -> 8.05 \times 10^{-6} | Helped |
| replace Adam with SOAP | PirateNet PINN · Burgers | relative L^2 error: 8.20 \times 10^{-5} -> 4.03\times 10^{-5} | Helped |
| replace Adam with SOAP | PirateNet PINN · Allen-Cahn | relative L^2 error: 2.24 \times 10^{-5} -> 3.48 \times 10^{-6} | Helped |
| replace Adam with SOAP | PirateNet PINN · Korteweg–De Vries | relative L^2 error: 7.04 \times 10^{-4} -> 3.40 \times 10^{-4} | Helped |
| replace Adam with SOAP | PirateNet PINN · Kuramoto-Sivashinsky | relative L^2 error: 7.48 \times 10^{-2} -> 3.86 \times 10^{-2} | Helped |
| replace Adam with SOAP | PirateNet PINN · Grey-Scott | relative L^2 error: 3.61 \times 10^{-3} -> 1.88 \times 10^{-4} | Helped |
| replace Adam with SOAP | PirateNet PINN · Ginzburg-Landau | relative L^2 error: 1.49 \times 10^{-2} -> 4.78 \times 10^{-3} | Helped |
| replace Adam with SOAP | PirateNet PINN · Lid-driven cavity (Re=5 × 10^3) | relative L^2 error: 3.24 \times 10^{-1} -> 3.99 \times 10^{-2} | Helped |
| replace Adam with SOAP | PirateNet PINN · Kolmogorov flow (Re=10^4) | relative L^2 error: 2.04 \times 10^{-1} -> 3.20 \times 10^{-2} | Helped |
| replace Adam with SOAP | PirateNet PINN · Rayleigh-Taylor instability (Ra=10^6) | relative L^2 error: 7.32 \times 10^{-2} -> 5.22 \times 10^{-3} | Helped |
| use SOAP instead of Adam (ablation) | PINN with standard MLP, modified MLP, or PirateNet · Wave, Burgers, Allen-Cahn, and KdV | relative L^2 error | Helped |
| increase SOAP preconditioner update frequency (ablation) | SOAP PINN · PDE benchmarks in the hyperparameter study | relative L^2 error | Helped |
Surprising findings
- Natural gradient descent in float64 fails on some random seeds and underperforms SOAP even when it succeeds, per the authors' Poisson and Heat experiments.
- ConFIG, a gradient-conflict method, showed sensitivity to hyperparameters and inconsistent performance in the authors' experiments.
Limitations
- SOAP requires about 2x longer training time than baselines.
- Theoretical intra-step analysis is restricted to a two-layer tanh network on 1D Laplace under small initialization.
- Several ablations and comparisons rely on the authors' own reimplementation of baselines.
Experimental setup
Datasets: Wave Burgers Allen-Cahn Korteweg-de Vries Kuramoto-Sivashinsky Grey-Scott Ginzburg-Landau Lid-driven cavity Kolmogorov flow Rayleigh-Taylor instability 2D Poisson 2D heat
Benchmarks: Wave Burgers Allen-Cahn Korteweg-de Vries Kuramoto-Sivashinsky Grey-Scott Ginzburg-Landau Lid-driven cavity Kolmogorov flow Rayleigh-Taylor instability
Metrics: relative L2 error gradient alignment score training time
Baselines: Adam Adam+L-BFGS Kron Muon ConFIG Natural gradient PCGrad IMTL-G
Builds on: PirateNet SOAP Shampoo Adam ConFIG
Closest papers by meaning
- Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks (NeurIPS 2026 · similarity 0.92)
- Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.91)
- ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks (ICLR 2025 · similarity 0.90)
- RoPINN: Region Optimized Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.90)
- Dual Cone Gradient Descent for Training Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.90)
- Sub-Sequential Physics-Informed Learning with State Space Model (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.