Research map / Optimization / Implicit bias and generalization
Optimization for Neural Operators can Benefit from Width
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves gradient descent convergence guarantees for Deep Operator Networks and Fourier Neural Operators via restricted strong convexity and smoothness, showing wider networks help optimization.
Problem
No optimization convergence guarantees exist for training Deep Operator Networks (DONs) and Fourier Neural Operators (FNOs) with gradient descent, despite universal approximation results.
Approach
A general GD optimization framework based on restricted strong convexity (RSC) and smoothness of the loss, applied to DONs and FNOs; the RSC constant and the neighborhood where guarantees hold both improve with network width.
Key results
- DON and FNO losses satisfy RSC and smoothness with high probability, giving GD loss decrease when the gradient norm is Omega(1/sqrt(m_f)+1/sqrt(m_g)) for DONs and Omega(1/sqrt(m)) for FNOs.
- Experiments on antiderivative, diffusion-reaction, and Burgers' problems with widths from 10 to 500 show wider DONs and FNOs reach lower training loss, e.g., around 1e-12 for wide DONs and 1e-5 for wide FNOs on the antiderivative operator.
- The RSC condition and the neighborhood radius both improve with width, since the RSC requirement weakens and the allowed radius scales with sqrt(m).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase width (ablation) | DON · Antiderivative | training loss | Helped |
| increase width (ablation) | DON · Diffusion-reaction | training loss | Helped |
| increase width (ablation) | DON · Burgers' equation | training loss | Helped |
| increase width (ablation) | FNO · Antiderivative | training loss | Helped |
| increase width (ablation) | FNO · Diffusion-reaction | training loss | Helped |
| increase width (ablation) | FNO · Burgers' equation | training loss | Helped |
| increase width (ablation) | DON · Antiderivative, diffusion-reaction, and Burgers' equation | optimization convergence speed | Helped |
| increase width (ablation) | FNO · Antiderivative, diffusion-reaction, and Burgers' equation | optimization convergence speed | Helped |
Limitations
- Analysis restricted to scalar input functions and smooth activations; vector-valued problems like Navier-Stokes not covered.
- Guarantees hold with probability 1-2KL(1/m_f+1/m_g) for DONs and 1-2(L+2)/m for FNOs and depend on assumed initialization scalings and sigma_0 constraints.
- Experiments only track training loss on canonical 1D problems, not generalization or large-scale applications.
Experimental setup
Datasets: antiderivative operator diffusion-reaction equation Burgers' equation
Benchmarks: antiderivative operator learning diffusion-reaction operator learning Burgers' equation operator learning
Metrics: empirical training loss
Builds on: Restricted strong convexity analysis of feedforward networks Deep Operator Networks Fourier Neural Operator
Closest papers by meaning
- Guaranteed Approximation Bounds for Mixed-Precision Neural Operators (ICLR 2024 · similarity 0.89)
- MgNO: Efficient Parameterization of Linear Operators via Multigrid (ICLR 2024 · similarity 0.88)
- Optimal deep learning of holomorphic operators between Banach spaces (NeurIPS 2024 · similarity 0.88)
- Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · similarity 0.87)
- Beyond Regular Grids: Fourier-Based Neural Operators on Arbitrary Domains (ICML 2024 · similarity 0.87)
- Amortized Fourier Neural Operators (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.