Research map / Optimization / Implicit bias and generalization
Understanding Optimization in Deep Learning with Central Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives "central flows," differential equations that model the time-averaged trajectory of oscillatory optimizers (gradient descent, scalar RMSProp, RMSProp) at the edge of stability, and shows they accurately predict long-term training trajectories of neural networks.
Problem
Traditional optimization theory cannot describe the dynamics of deterministic deep learning optimizers, which operate in an oscillatory 'edge of stability' regime where the sharpness hovers around the critical threshold 2/η.
Approach
Time-average the optimizer's trajectory via a third-order Taylor expansion; the oscillation covariance Σ(t) acts as an implicit curvature penalty, and it is determined uniquely by a semidefinite complementarity problem that keeps top Hessian eigenvalues at 2/η. The resulting flow is a projected gradient flow onto the stable region and tracks the smoothed optimizer trajectory.
Key results
- Central flow tracks the long-term gradient descent trajectory with small weight-space distance, while gradient flow distance grows large over time (ViT on CIFAR-10).
- The central flow's Sigma(t) accurately predicts the instantaneous variance of gradient descent oscillations along each unstable Hessian eigenvector.
- Central flows accurately predict time-averaged training loss curves for gradient descent and RMSProp across several architectures and learning rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the central flow instead of gradient flow to model gradient descent at the edge of stability | Gradient descent trajectory modeling · Deep learning settings (various architectures) | long-term trajectory prediction accuracy | Helped |
| use the central flow to predict the oscillation covariance | Gradient descent · Vision Transformer on CIFAR-10 subset | oscillation covariance prediction accuracy | Helped |
| use the central flow to predict the smoothed training loss | Gradient descent · Deep learning settings (various architectures) | time-averaged training loss prediction accuracy | Helped |
| increase the learning rate | Central flow approximation of gradient descent · Deep learning settings | trajectory prediction accuracy | Hurt |
| use cross-entropy rather than MSE loss | Central flow approximation of gradient descent · Deep learning problems | trajectory prediction accuracy | Hurt |
Surprising findings
- The loss along the actual gradient descent trajectory is consistently higher than along the central flow, because oscillations bounce between valley walls.
- Central flow's loss decrease is strictly slower than that of gradient flow, since oscillation-induced curvature reduction removes gradient components available for descent.
Limitations
- Derivation is informal/heuristic; no proof that discrete optimizers follow the central flow.
- Accuracy degrades at large learning rates and with cross-entropy loss due to higher-order effects and spikes.
- Analysis restricted to deterministic full-batch training; stochastic extension is open.
Experimental setup
Datasets: CIFAR-10
Metrics: distance between gradient descent and central flow weights time-averaged training loss variance of oscillations predicted by sigma^2 / Sigma(t)
Baselines: gradient flow modified gradient flow stochastic differential equation models
Builds on: gradient flow edge of stability third-order Taylor analysis of EOS (Damian et al.) RMSProp Adam
Closest papers by meaning
- Continuous vs. Discrete Optimization of Deep Neural Networks (NeurIPS 2021 · similarity 0.90)
- On the Convergence Direction of Gradient Descent (ICLR 2026 · similarity 0.89)
- Investigation into Training Dynamics of Learned Optimizers (Student Abstract) (AAAI 2024 · similarity 0.87)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.87)
- Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.