Research map / Learning theory and algorithms / Kernels and regression theory
Understanding the Expressivity and Trainability of Fourier Neural Operator: A Mean-Field Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a mean-field theory for random Fourier Neural Operators, showing ordered/chaotic phases and an edge-of-chaos initialization that governs trainability, and validates it on PDE benchmarks.
Problem
Lack of theoretical understanding of the expressivity and trainability (gradient stability) of the Fourier Neural Operator, and why deep FNOs train unstably.
Approach
Applies mean-field theory (ordered-chaos phase transition, edge of chaos) to a random FNO, showing mode truncation induces FNO-specific covariance dynamics whose gradient behavior is governed by chi_c*, yielding an edge-of-chaos initialization (He-like, sigma^2=2 for ReLU).
Key results
- Shows an edge-of-chaos transition: gradients vanish when chi_c* < 1 and explode when chi_c* > 1, with sigma^2 around 2 needed for stable training of deep simplified FNO with ReLU/Tanh.
- Experiments across advection, Burgers', Darcy, and Navier-Stokes show that as depth L increases, the acceptable range of initial sigma^2 narrows and sigma^2 near 2 is required for stable training.
- Derives FNO He-initialization (Theta, Xi ~ N(0, D^-1)) for ReLU that preserves gradient norm in deep simplified FNO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize near the edge of chaos (σ² ≈ 2) | deep FNO · advection, Burgers', Darcy Flow, and Navier-Stokes | training stability | Helped |
| increase depth while keeping initialization away from the edge of chaos | FNO · advection, Burgers', Darcy Flow, and Navier-Stokes | training stability | Hurt |
| set σ² below approximately 2 with σ²_b = 0.1 | simplified FNO with Tanh · gradient backpropagation | gradient norm | Hurt |
| set σ² above approximately 2 with σ²_b = 0.1 | simplified FNO with Tanh · gradient backpropagation | gradient norm | Hurt |
| use σ² = 2 and zero bias (He initialization) | simplified FNO with ReLU · deep-network backpropagation | gradient norm | Helped |
Limitations
- Analysis is restricted to the network at initialization and does not address the full optimization dynamics
- Provides only a necessary condition for stable training, not sufficient conditions
- Dropout, batch normalization, and skip connections are not analyzed
Experimental setup
Datasets: advection equation Burgers' equation Darcy flow Navier-Stokes
Benchmarks: advection equation Burgers' equation Darcy flow Navier-Stokes
Metrics: training MSE normalized MSE
Baselines: Fourier Neural Operator
Builds on: Fourier Neural Operator mean-field theory for deep networks edge of chaos initialization He initialization
Closest papers by meaning
- Amortized Fourier Neural Operators (NeurIPS 2024 · similarity 0.86)
- Component Fourier Neural Operator for Singularly Perturbed Differential Equations (AAAI 2024 · similarity 0.85)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.85)
- Beyond Regular Grids: Fourier-Based Neural Operators on Arbitrary Domains (ICML 2024 · similarity 0.85)
- Optimization for Neural Operators can Benefit from Width (ICML 2025 · similarity 0.84)
- Maximum Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.