Research map / Optimization / Optimizers
Dual Cone Gradient Descent for Training Physics-Informed Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dual Cone Gradient Descent (DCGD), a gradient-manipulation optimizer that keeps the update inside the dual cone of the PDE-residual and boundary-loss gradients, to fix conflicting and dominating gradients when training PINNs.
Problem
PINNs often fail to converge to correct solutions because the PDE residual and boundary loss gradients conflict (negative inner product) or differ greatly in magnitude, so minimizing the total loss can increase one of the component losses.
Approach
Derives a necessary and sufficient geometric condition (angle and magnitude ratio) for the total gradient to lie in the dual cone of the two loss gradients, and defines DCGD variants (projection, average, center) that always choose update directions inside that dual cone; the center variant projects the total gradient onto the angle bisector of the two loss gradients.
Key results
- On the Helmholtz benchmark, DCGD reaches mean relative L2 error 0.0029 vs 0.0609 for Adam and 0.0066 for LRA (best of the baselines in that column).
- DCGD achieves the lowest mean error on Burgers' (0.0124), Klein-Gordon (0.0069, tied with LRA's 0.0069), Heat 5D (0.0008) and Helmholtz 3D (0.0774), while Adam fails at 0.6109 on Helmholtz 3D.
- On the convection equation, DCGD reduces relative L2 error from 0.4880 (baseline) to 0.0246.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DCGD instead of LRA | vanilla PINN · Helmholtz | relative L^2 error: 0.0066 (0.0025) -> 0.0029 (0.0005) | Helped |
| use DCGD instead of PCGrad | vanilla PINN · Burgers' | relative L^2 error: 0.0159 (0.0061) -> 0.0124 (0.0046) | Helped |
| use DCGD instead of LRA | vanilla PINN · Klein-Gordon | relative L^2 error: 0.0069 (0.0037) -> 0.0069 (0.0027) | No clear effect |
| use DCGD instead of MultiAdam | vanilla PINN · Heat (5D) | relative L^2 error: 0.0009 (0.0007) -> 0.0008 (0.0003) | Helped |
| use DCGD instead of LRA | vanilla PINN · Helmholtz (3D) | relative L^2 error: 0.0831 (0.0123) -> 0.0774 (0.0250) | Helped |
| combine DCGD with LRA | vanilla PINN · Helmholtz, Burgers', Klein-Gordon, Heat (5D), and Helmholtz (3D) | relative L^2 error: 0.0029 (0.0005) -> 0.0023 (0.0007); 0.0124 (0.0046) -> 0.0104 (0.0021); 0.0069 (0.0027) -> 0.0050 (0.0013); 0.0008 (0.00 | Mixed |
| combine DCGD with NTK | vanilla PINN · Helmholtz, Burgers', Klein-Gordon, Heat (5D), and Helmholtz (3D) | relative L^2 error: 0.0029 (0.0005) -> 0.0057 (0.0035); 0.0124 (0.0046) -> 0.0113 (0.0040); 0.0069 (0.0027) -> 0.0055 (0.0014); 0.0008 (0.00 | Mixed |
| apply DCGD (Center) | causal PINN · Chaotic KS | relative L^2 error: 0.0687 -> 0.0376 | Helped |
| apply DCGD (Center) | PINNsFormer · Convection | relative L^2 error: 0.4880 -> 0.0246 | Helped |
| apply DCGD (Center) | A-PINN · Volterra IDEs | relative L^2 error: 0.0068 -> 0.0011 | Helped |
Surprising findings
- Conflicting gradients (negative inner product) occur in about half of training iterations for the Helmholtz PINN, and the PDE-residual gradient is several tens to hundreds of times larger than the boundary gradient.
- MGDA fails badly on PINN benchmarks (e.g., 0.7590 error on Helmholtz) and does not converge on Heat 5D.
Limitations
- Only three variants (projection, average, center) are studied; the authors note a more powerful dual-cone specification could exist.
- Multi-objective setting is restricted to two losses (PDE residual and boundary); more general multi-task problems are left to future work.
- Convergence guarantee relies on a conflict threshold and an assumed L-smooth total loss with Lipschitz gradient; the analysis is non-convex but depends on these assumptions.
Experimental setup
Datasets: Helmholtz equation viscous Burgers' equation Klein-Gordon equation 5D heat equation 3D Helmholtz equation chaotic Kuramoto-Sivashinsky equation convection equation Volterra integro-differential equations double pendulum
Benchmarks: Helmholtz Burgers' Klein-Gordon Heat (5D) Helmholtz (3D) Chaotic KS Convection Volterra IDEs double pendulum
Metrics: relative L2 error standard deviation max min
Baselines: Adam LRA NTK PCGrad MGDA CAGrad Aligned-MTL MultiAdam DPM
Builds on: PINNs PCGrad Learning rate annealing Neural Tangent Kernel weighting Adam
Closest papers by meaning
- Challenges in Training PINNs: A Loss Landscape Perspective (ICML 2024 · similarity 0.90)
- Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective (NeurIPS 2025 · similarity 0.90)
- RoPINN: Region Optimized Physics-Informed Neural Networks (NeurIPS 2024 · similarity 0.90)
- ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks (ICLR 2025 · similarity 0.89)
- PINNacle: A Comprehensive Benchmark of Physics-Informed Neural Networks for Solving PDEs (NeurIPS 2024 · similarity 0.89)
- Learning a Neural Solver for Parametric PDEs to Enhance Physics-Informed Methods (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.