Research map / Optimization / Implicit bias and generalization
Leveraging the two-timescale regime to demonstrate convergence of neural networks
Pierre Marion, Raphaël Berthier
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use two-timescale regime (inner-layer step size much smaller than outer-layer, epsilon = 2e-5) instead of epsilon = 1 (ablation) | one-pass SGD on shallow sigmoid network, m moderate · 1D piecewise-constant target recovery | recovery of target discontinuities (qualitative) | Improved |
| set epsilon = 1 so all neurons move toward discontinuities (ablation) | SGD, shallow network · 1D piecewise-constant target | coverage of discontinuities by neurons | Hurt |
| use two-timescale regime, giving sparse representation where only neurons nearest discontinuities move | SGD, shallow network · 1D piecewise-constant target | number of active neurons | Improved |
| compare SGD with eta = 4e-3 to gradient flow in two-timescale limit with eta = 0 | SGD, two-timescale regime · 1D piecewise-constant target, loss vs steps | loss L | No clear effect |
| nonzero eta (sharpness) in SGD versus eta = 0 gradient flow | SGD vs two-timescale limit · 1D piecewise-constant target | final error | Hurt |
| apply two-timescale regime to d-dimensional cuboid piecewise-constant targets (d=2, 10) (ablation) | SGD, shallow network · 2D/10D piecewise-constant target | coverage of discontinuities | Improved |
| use ReLU activations with piecewise-affine targets under two-timescale regime | SGD, shallow ReLU network · piecewise-affine target | recovery (qualitative) | Improved |
Closest papers by meaning
- Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks (NeurIPS 2022 · similarity 0.90)
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.90)
- The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (ICML 2025 · similarity 0.89)
- Uniform-in-Time Weak Propagation of Chaos in Shallow Neural Networks (NeurIPS 2026 · similarity 0.89)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.88)
- On the Unstable Convergence Regime of Gradient Descent (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
