Research map / Optimization / Implicit bias and generalization
Gradient flow and mean-field dynamics
Studies training dynamics of wide networks via gradient flow, mean-field limits, and teacher-student models.
77 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Implicit bias and generalization, keyword cluster “relu · width · kernel”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
19 papers in 2023–24 and 27 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 8.1% to 10.3%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks (NeurIPS 2018 · 52 citations)
- 2019 · Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent (NeurIPS 2019 · 290 citations)
- 2020 · Asymptotics of Wide Networks from Feynman Diagrams (ICLR 2020 · 51 citations)
- 2021 · A Recipe for Global Convergence Guarantee in Deep Neural Networks (AAAI 2021 · 8 citations)
- 2022 · High-dimensional limit theorems for SGD: Effective dynamics and critical scaling (NeurIPS 2022 · 18 citations)
- 2023 · Global Convergence Analysis of Local SGD for Two-layer Neural Network without Overparameterization (NeurIPS 2023 · 2 citations)
- 2024 · Exact, Fast and Expressive Poisson Point Processes via Squared Neural Families (AAAI 2024 · 2 citations)
- 2025 · Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks (NeurIPS 2025 · 1 citation)
- 2026 · Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks (ICML 2026 · 0 citations)
Earliest papers here
- Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks (NeurIPS 2018 · 52 citations)
- Mean Field for the Stochastic Blockmodel: Optimization Landscape and Convergence Issues (NeurIPS 2018 · 12 citations)
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent (NeurIPS 2019 · 290 citations)
What has been tried
48 papers here report 292 measured changes (a component added, removed or swapped, compared with a baseline): helped 56%, no clear effect 15%, hurt 19%, mixed 10%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add positional encoding to the attention model | SSI target with SIE = 4 · sequence information exponent | 4 -> 2 | Helped | Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networks (NeurIPS 2025 · 1 citation) |
| Use DANA-decaying instead of DANA-constant momentum | PLRF, 2α > 1 and 0 < ℓ < 2α · loss exponent | Helped | Dimension-adapted Momentum Outscales SGD (NeurIPS 2025 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 20 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 2 of 20 papers
Baselines
- gradient descent · 3 of 20 papers
- Adam · 2 of 20 papers
- SGD · 2 of 20 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 46 papers; from OpenAlex and the papers' affiliations).
- University of Toronto · 5 papers
- New York University · 4 papers
- The University of Tokyo · 3 papers
- EPFL · 3 papers
- Vector Institute · 3 papers
- Kyushu Institute of Technology · 2 papers
- Northwestern University · 2 papers
- The University of Hong Kong · 2 papers
- Università di Pisa · 2 papers
- University of California, San Diego · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Shallow ReLU network convergence (132 papers)
- Neural tangent kernel analysis (114 papers)
- Low-rank and single-index recovery (70 papers)
- In-context learning as gradient descent (41 papers)
Elsewhere in the topic
- Edge-of-stability dynamics (116 papers)
- SGD noise and flat minima (112 papers)
- Stability-based generalization analysis (88 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- Sharpness-aware minimization theory (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).