Research map / Architectures and efficiency / Pruning and efficient tuning
Maximum Update Parametrization and Zero-Shot Hyperparameter Transfer for Fourier Neural Operators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a Maximal Update Parametrization for Fourier Neural Operators under scaling of Fourier modes K and proposes μTransfer-FNO, which tunes hyperparameters on small FNOs and transfers them zero-shot to billion-parameter FNOs.
Problem
Hyperparameter tuning becomes computationally infeasible for Fourier Neural Operators when scaling up the number of Fourier modes K, since parameters grow as O(K^d) and optimal hyperparameters shift across scales under standard parametrization.
Approach
Extends the abc-parametrization/muP theory to the FNO kernel integral operator under Fourier-mode scaling, deriving init-variance and learning-rate scaling Theta(1/sqrt(d log K)) via a maximum-of-sub-Gaussians argument, so that small-K proxy hyperparameters transfer to large-K FNOs.
Key results
- On Navier-Stokes with FNO-3D, muTransfer-FNO reaches 5.34% L2 relative error vs 5.69% for direct tuning of the full model, using 0.30x total training cost.
- On Darcy Flow with FNO-2D, muTransfer-FNO achieves 1.22% test error vs 1.25% for full-model tuning, but at 1.38x the training cost.
- Under standard parametrization the optimal learning rate shifts (e.g., 1.8e-3 at K=3 to 7.4e-4 at K=24 on Navier-Stokes), while under muTransfer it stays approximately stable across K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use μTransfer-FNO instead of directly tuning the full model | FNO-2D · Darcy Flow | L2 relative error: 1.25% -> 1.22% | Helped |
| Use μTransfer-FNO instead of directly tuning the full model | FNO-2D · Darcy Flow | total training cost: 1× -> 1.38× | Hurt |
| Use μTransfer-FNO instead of directly tuning the full model | FNO-3D · Navier-Stokes Equation | L2 relative error: 5.69% -> 5.34% | Helped |
| Use μTransfer-FNO instead of directly tuning the full model | FNO-3D · Navier-Stokes Equation | total training cost: 1× -> 0.30× | Helped |
| Apply μTransfer-FNO parametrization when scaling Fourier modes | FNO-3D · Navier-Stokes Equation | optimal learning rate stability across model sizes: 1.8×10^{-3} -> 7.4×10^{-4} under standard parametrization; approximately 4.2×10^{-3} under μTransfer-FNO | Helped |
| Apply μTransfer-PINO parametrization when scaling Fourier modes | PINO-2D · Darcy Flow | optimal learning rate stability across model sizes: approximately 5.6×10^{-3} | Helped |
| Apply μTransfer-FNO to batch-size selection across model sizes | FNO-2D · Darcy Flow | optimal batch size stability across model sizes: 20 | Helped |
| Apply μTransfer-FNO to Adam β2 selection across model sizes | FNO-2D · Darcy Flow | optimal β2 stability across model sizes: 0.98 | Helped |
Surprising findings
- On Darcy Flow, muTransfer-FNO was more expensive (1.38x) than naive full-model tuning because the cost gap between small and large models is small at that scale.
Limitations
- Results are restricted to FNO and do not cover other neural operator variants such as DeepONet, MGNO, or multi-wavelet NO.
- Optimal learning rate sometimes deviates at very small K, attributed to training randomness.
- Theory assumes tanh or sigma-GELU activations, Adam optimizer, and sub-Gaussian updates enforced via element-wise gradient clipping.
Experimental setup
Datasets: 1D Burgers' Equation 2D Darcy Flow 2D incompressible Navier-Stokes (vorticity form)
Benchmarks: Burgers' Equation Darcy Flow Navier-Stokes Equation
Metrics: L2 relative error training loss total training cost
Baselines: Standard parametrization (tune full model)
Builds on: Maximal Update Parametrization muTransfer Fourier Neural Operator Physics-Informed Neural Operator
Closest papers by meaning
- Amortized Fourier Neural Operators (NeurIPS 2024 · similarity 0.91)
- Guaranteed Approximation Bounds for Mixed-Precision Neural Operators (ICLR 2024 · similarity 0.89)
- Component Fourier Neural Operator for Singularly Perturbed Differential Equations (AAAI 2024 · similarity 0.88)
- Light-Weight Diffusion Multiplier and Uncertainty Quantification for Fourier Neural Operators (NeurIPS 2025 · similarity 0.88)
- Beyond Regular Grids: Fourier-Based Neural Operators on Arbitrary Domains (ICML 2024 · similarity 0.88)
- Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.