Research map / Probabilistic and scientific modeling / Differential equations and PDEs
Heavy-Ball Momentum Method in Continuous Time and Discretization Error Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a piecewise ODE (HB Flow) for discrete heavy-ball momentum whose discretization error is controlled to arbitrary order in the step size, and applies it to directional smoothness and diagonal linear network implicit bias.
Problem
Continuous-time approximations of heavy-ball momentum (HB) via rescaled gradient flow have only second-order-or-worse discretization error, so they fail to distinguish HB from GD in continuous time.
Approach
Derives a piecewise first-order ODE (HB Flow) with counter terms gamma_k obtained by matching Taylor-expansion residuals order by order in the step size, so discretization error can be made O(eta^alpha) for any alpha; applies it to implicit regularization of directional smoothness and implicit bias of diagonal linear networks.
Key results
- HB with mu=0.9 and eta=0.1 on MLP/CIFAR-10 keeps directional smoothness below GD's saturation level of about 2/eta=20 and decreasing.
- The O(eta^3) HBF trajectory is closer to discrete HB than RGF and the O(eta^2) HBF in a 2-d model, with the lowest discretization error after several steps.
- The O(eta^2) HBF implies an implicit gradient regularization scaled by (1+mu)/(1-mu)^3 relative to GD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HB instead of GD | MLP · CIFAR-10 | directional smoothness | Helped |
| use HBF with discretization error O(η³) instead of RGF | HB continuous-time approximation · 2-d model | trajectory agreement with discrete HB | Helped |
| use HBF with discretization error O(η³) instead of HBF with O(η²) (ablation) | HB continuous-time approximation · 2-d model | discretization error | Helped |
| use HBF instead of RGF | HB continuous-time approximation · 2-d model | trajectory agreement with discrete HB | Helped |
| add the HBF high-order correction terms | diagonal linear network trained with HB · interpolation solution | implicit-bias characterization | Mixed |
Surprising findings
- Rescaled gradient flow yields the same limit points as GF, so it cannot distinguish HB from GD, whereas the higher-order HBF can.
Limitations
- Does not cover adaptive methods such as Adam.
- Analysis is deterministic full-batch; stochastic case left as future work.
- Only diagonal linear networks are studied in the deep learning application.
Experimental setup
Datasets: CIFAR-10 synthetic 2-d regression
Benchmarks: CIFAR-10
Metrics: directional smoothness discretization error norm trajectory comparison
Baselines: rescaled gradient flow gradient flow second-order ODE for heavy-ball GD continuous approximation with O(eta^2) error
Builds on: rescaled gradient flow backward error analysis implicit gradient regularization Taylor expansion residual matching
Closest papers by meaning
- Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games (ICML 2025 · similarity 0.87)
- Accelerated Convergence of Stochastic Heavy Ball Method under Anisotropic Gradient Noise (ICLR 2024 · similarity 0.86)
- Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance (ICLR 2025 · similarity 0.86)
- Accelerated Over-Relaxation Heavy-Ball Method: Achieving Global Accelerated Convergence with Broad Generalization (ICLR 2025 · similarity 0.86)
- Effects of Momentum in Implicit Bias of Gradient Flow for Diagonal Linear Networks (AAAI 2025 · similarity 0.86)
- Nonlinearly Preconditioned Gradient Methods: Momentum and Stochastic Analysis (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.