Research map / Optimization / Implicit bias and generalization
Global Convergence in Neural ODEs: Impact of Activation Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that smooth, nonlinear non-polynomial activations make Neural ODE forward/backward dynamics well-posed and their NTK converge and stay strictly positive definite, yielding global linear convergence of gradient descent in overparameterized regimes.
Problem
Training Neural ODEs with gradient descent lacks theoretical guarantees: gradient computation accuracy via numerical ODE solvers, well-posedness of forward/backward ODEs, and global convergence analysis (NTK positive definiteness) are not established for continuous-depth models.
Approach
Approximates Neural ODEs by sequences of finite-depth ResNets, uses Moore-Osgood theorem to interchange depth and width limits for NNGP and NTK, derives an integral form of the limiting NTK, and shows SPD holds for non-polynomial activations when phi and phi' are Lipschitz.
Key results
- Output and gradient difference between Neural ODE and finite-depth ResNet decays as roughly 1/L (L from 10 to 1,000) with Softplus activation, uniformly across widths.
- NTK smallest eigenvalue stays strictly positive during training for widths above the training set size (1000 samples).
- Theorem gives linear loss decay L(theta_k) <= (1 - eta*lambda_0/16)^k L(theta_0) under Lipschitz phi and phi' with non-polynomial nonlinearity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Softplus instead of ReLU (ablation) | Neural ODE · Neural ODE training | convergence speed | Helped |
| scale the dynamics with σ_w ∼ 1/T (ablation) | Neural ODE · long-time-horizon training | training stability | Helped |
| increase the width (ablation) | Neural ODE · gradient descent training | convergence speed | Helped |
| increase the width (ablation) | Neural ODE · NTK during training | smallest NTK eigenvalue | Helped |
| increase the width (ablation) | Neural ODE · test set | test loss | Helped |
| use quadratic activation instead of a non-polynomial activation (ablation) | Neural ODE · NTK | strict positive definiteness | Mixed |
| use a quadratic activation (ablation) | Neural ODE · training | convergence speed | Hurt |
| increase the finite-depth ResNet approximation depth (ablation) | Neural ODE · MNIST | output approximation error: 1/L | Helped |
| increase the finite-depth ResNet approximation depth (ablation) | Neural ODE · MNIST | gradient approximation error: 1/L | Helped |
| increase network width (ablation) | Neural ODE · NTK approximation to limiting NTK | NTK convergence: between 1/m and 1/√m | Helped |
| use width greater than the number of training samples (ablation) | Neural ODE · NTK during training | smallest NTK eigenvalue | Helped |
| use width smaller than the number of training samples (ablation) | Neural ODE · NTK during training | smallest NTK eigenvalue | Hurt |
Surprising findings
- Quadratic (polynomial) activations also yield SPD NTK in experiments, though with slower convergence, despite theory requiring non-polynomial activations.
- Improper time scaling (sigma_w not ~1/T) causes early-stage damping in long-horizon Neural ODE training.
Limitations
- NTK convergence rate to the limit is shown only empirically; no theoretical rate given for width.
- Experiments are on small-scale simple Neural ODE architectures with a single hidden-state form and fixed-width settings.
Experimental setup
Datasets: MNIST CIFAR-10 AG News Daily Climate
Benchmarks: MNIST CIFAR-10 AG News Daily Climate
Metrics: training loss test loss NTK smallest eigenvalue output and gradient approximation error
Baselines: ReLU finite-depth ResNet
Builds on: Neural ODE neural tangent kernel NNGP dual activation Tensor programs
Closest papers by meaning
- How Deep Do We Need: Accelerating Training and Inference of Neural ODEs via Control Perspective (ICML 2024 · similarity 0.89)
- How to Train Your Neural ODE: the World of Jacobian and Kinetic Regularization (ICML 2020 · similarity 0.89)
- Do Residual Neural Networks discretize Neural Ordinary Differential Equations? (NeurIPS 2022 · similarity 0.88)
- Semi-Implicit Neural Ordinary Differential Equations (AAAI 2025 · similarity 0.87)
- Global convergence of adjoint-optimized neural PDEs (NeurIPS 2026 · similarity 0.87)
- ControlSynth Neural ODEs: Modeling Dynamical Systems with Guaranteed Convergence (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.