Research map / Learning theory and algorithms / Prediction and decision losses
Position: Optimization in SciML Should Employ the Function Space Geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that SciML optimizers should follow "first optimize, then discretize" using function-space geometry, showing that several state-of-the-art SciML methods (ENGD, CPINN/CGD, GN natural gradient, VMC natural gradient) arise as discretized infinite-dimensional algorithms.
Problem
Optimization for neural-network-based scientific machine learning (PINNs, deep Ritz, variational Monte Carlo, neural operators) is notoriously hard, with first-order methods plateauing and optimization error dominating; principled optimizer design is lacking.
Approach
Choose an infinite-dimensional algorithm (gradient descent, Newton, Gauss-Newton, Lagrange-Newton in a Hilbert space) based on the continuous PDE-derived formulation, then discretize it in the tangent space of the neural network ansatz; this yields a Gramian-preconditioned parameter update whose function-space update is the orthogonal projection onto the model's tangent space up to quadratic error
Key results
- For a 2D Poisson PINN, first-order optimizers (Adam, GD) plateau while ENGD and Newton perform better, and function-space ENGD reaches accuracy several orders of magnitude higher (as reported in the paper's figure).
Surprising findings
- Applying Newton's method directly to the discretized parameter-space PINN loss does not reproduce the infinite-dimensional Newton/Lagrange-Newton algorithm, because second derivatives of the network do not vanish.
Limitations
- Scalability: solving linear systems of size the parameter count is intractable for large networks without matrix-free methods or K-FAC.
- Non-symmetric or indefinite T_u lacks the orthogonal projection guarantee and needs inf-sup-type conditions.
- Argument is largely conceptual; no new algorithm or large-scale benchmark is introduced.
Experimental setup
Metrics: training loss L2 error
Baselines: Adam gradient descent Newton's method K-FAC
Builds on: natural gradient descent energy natural gradient descent competitive gradient descent K-FAC deep Ritz method
Closest papers by meaning
- Optimization Can Learn Johnson Lindenstrauss Embeddings (NeurIPS 2024 · similarity 0.83)
- Bayesian Optimization on Function Spaces via Sparse RKHS Manifolds (NeurIPS 2026 · similarity 0.82)
- Automated Kernel Discovery Towards Understanding High-dimensional Bayesian Optimization (NeurIPS 2026 · similarity 0.81)
- Piecewise Constant and Linear Regression Trees: An Optimal Dynamic Programming Approach (ICML 2024 · similarity 0.81)
- What Data Enables Optimal Decisions? An Exact Characterization for Linear Optimization (NeurIPS 2025 · similarity 0.81)
- Vanilla Bayesian Optimization Performs Great in High Dimensions (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.