Research map / Optimization / Implicit bias and generalization
Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes conservation laws for momentum (second-order) flows in Euclidean and non-Euclidean geometries, showing they are time/velocity dependent and fewer than for gradient flows across linear, ReLU, NMF and ICNN models.
Problem
Conservation laws (quantities preserved during training) are well understood for Euclidean gradient flows but largely unknown for momentum-based dynamics and non-Euclidean (mirror/natural gradient) metrics.
Approach
Define conserved functions of time, parameters and velocity via phase-space lifting; show that momentum conservation laws have time dependence (via exp(tau t) for heavy ball), reduce to a Lie-algebra characterization of the trace of a vector-field space, and yield a systematic loss of conservation relative to gradient flow.
Key results
- For two-layer linear networks with Euclidean momentum flow, the number of independent conservation laws is rk/2·(2r-1-rk), and for r ≤ n+m the momentum case has N_GF − N_MF = r fewer conservation laws than gradient flow.
- For two-layer ReLU networks (without bias) under Euclidean momentum flow, no conservation law exists.
- For NMF with mirror geometry, gradient flow has r independent conservation laws (1ᵀU − 1ᵀV), whereas the non-Euclidean momentum flow has none.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add momentum to Euclidean gradient flow | Two-layer linear networks with r ≤ n+m · Euclidean geometry | number of independent conservation laws: N_GF-N_MF = r > 0 | Hurt |
| add momentum to Euclidean gradient flow | Two-layer linear networks with r > n+m · Euclidean geometry | number of independent conservation laws: N_GF-N_MF ≤ 0 | Mixed |
| add momentum | Two-layer ReLU networks · Euclidean geometry | number of conservation laws | Hurt |
| add momentum to mirror gradient flow | NMF · Shannon entropy mirror geometry | number of conservation laws: r -> 0 | Hurt |
| add momentum to hybrid gradient flow | ICNN · Hybrid mirror/Euclidean geometry | number of conservation laws | Hurt |
| use natural gradient instead of Euclidean gradient flow | Neural networks with known Euclidean gradient-flow conservation laws · Natural gradient flow | number of known conservation laws preserved | No clear effect |
Surprising findings
- Momentum flows have no conservation laws for ReLU networks and for NMF/ICNN in the non-Euclidean momentum case, despite gradient flows having them.
- Momentum conservation laws depend non-trivially on time (e.g., through exp(tau t)), unlike gradient flow laws.
Limitations
- Theory restricted to continuous-time flows; discretization effects not covered by the theory.
- Metric M must not depend on the dataset for the main momentum characterization; natural gradient only partially treated.
- Deeper-network completeness is established empirically via computations on small sizes rather than proved.
Experimental setup
Metrics: number of independent conservation laws
Builds on: Noether's theorem Marcotte et al. 2023 conservation laws for gradient flows (Lie algebra character Polyak heavy ball Nesterov acceleration ODE mirror descent
Closest papers by meaning
- Transformative or Conservative? Conservation laws for ResNets and Transformers (ICML 2025 · similarity 0.92)
- SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures (NeurIPS 2025 · similarity 0.86)
- A geometric framework for momentum-based optimizers for low-rank training (NeurIPS 2025 · similarity 0.86)
- Riemann Tensor Neural Networks: Learning Conservative Systems with Physics-Constrained Networks (ICML 2025 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.