Research map / Optimization / Implicit bias and generalization
Low-rank and single-index recovery
Analyzes gradient descent on matrix sensing, factorization, and single-index models with global guarantees.
70 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Implicit bias and generalization, keyword cluster “relu · width · kernel”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 21 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 9.4% to 8.0%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Global Optimality of Local Search for Low Rank Matrix Recovery (NeurIPS 2016 · 241 citations)
- 2017 · A Riemannian Network for SPD Matrix Learning (AAAI 2017 · 378 citations)
- 2018 · On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport (NeurIPS 2018 · 173 citations)
- 2019 · Subgradient Descent Learns Orthogonal Dictionaries (ICLR 2019 · 35 citations)
- 2020 · Agnostic Learning of a Single Neuron with Gradient Descent (NeurIPS 2020 · 15 citations)
- 2021 · Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization (NeurIPS 2021 · 6 citations)
- 2022 · Sharp Restricted Isometry Property Bounds for Low-Rank Matrix Recovery Problems with Corrupted Measurements (AAAI 2022 · 5 citations)
- 2023 · The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensing (ICML 2023 · 10 citations)
- 2024 · On the hardness of learning under symmetries (ICLR 2024 · 1 citation)
- 2025 · Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · 0 citations)
- 2026 · Data Reconstruction: Identifiability and Optimization with Sample Splitting (ICML 2026 · 0 citations)
Earliest papers here
- Global Optimality of Local Search for Low Rank Matrix Recovery (NeurIPS 2016 · 241 citations)
- Low-rank Solutions of Linear Matrix Equations via Procrustes Flow (ICML 2016 · 139 citations)
- Globally Optimal Training of Generalized Polynomial Neural Networks with Nonlinear Spectral Methods (NeurIPS 2016 · 12 citations)
What has been tried
53 papers here report 282 measured changes (a component added, removed or swapped, compared with a baseline): helped 63%, no clear effect 12%, hurt 19%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add one extra parameterization layer (D=1) | Sequence model (p=1, q=2, gamma=1.1) · convergence rate | 0.48 -> 0.49 | Helped | Improving Adaptivity via Over-Parameterization in Sequence Models (NeurIPS 2024 · 1 citation) |
| Modify the noise spectrum while leaving the problem unregularized | Alignment mismatch ablation · generalization risk | No clear effect | Least Squares Regression Can Exhibit Under-Parameterized Double Descent (NeurIPS 2024 · 1 citation) | |
| replace exponentially decaying step sizes with constant step size and averaged iterates | High-dimensional and asymptotic settings · excess risk | \frac{1}{n^2\eta_0^s} | Hurt | Learning Curves of Stochastic Gradient Descent in Kernel Regression (ICML 2025 · 0 citations) |
| increase width | Antiderivative · training loss | Helped | Optimization for Neural Operators can Benefit from Width (ICML 2025 · 0 citations) | |
| Use a learning rate of 10^{-3} instead of 10^{-4} or smaller | Synthetic optimization, T=20 · relative objective improvement | Hurt | Learning Provably Improves the Convergence of Gradient Descent (NeurIPS 2025 · 0 citations) | |
| Perturb Nyström initialization by adding residual-space noise | Symmetric exact-parametrized matrix factorization · convergence rate | quadratic -> linear | Hurt | On the Crucial Role of Initialization for Matrix Factorization (ICLR 2025 · 0 citations) |
| Set the learning rate to 0.0025 rather than 0.001 or 0.005 | Vandermonde V · probability of r reaching approximately 1 | Helped | Symmetric Single Index Learning (ICLR 2024 · 0 citations) | |
| Use the tailor-designed initialization instead of RI initialization | ReLU MC, σ = 0 · completion error γ_T | 0.5 ± 0.13 -> (7.6 ± 0.7) · 10^{-11} | Helped | Symmetric Matrix Completion with ReLU Sampling (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 12 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- kernel ridge regression · 2 of 12 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 43 papers; from OpenAlex and the papers' affiliations).
- The University of Tokyo · 4 papers
- New York University · 3 papers
- University of Washington · 3 papers
- University of California, San Diego · 3 papers
- University of Michigan · 2 papers
- Massachusetts Institute of Technology · 2 papers
- Tsinghua University · 2 papers
- University of Chicago · 2 papers
- University of Michigan - Ann Arbor · 2 papers
- Toyota Technological Institute at Chicago · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Shallow ReLU network convergence (132 papers)
- Neural tangent kernel analysis (114 papers)
- Gradient flow and mean-field dynamics (77 papers)
- In-context learning as gradient descent (41 papers)
Elsewhere in the topic
- Edge-of-stability dynamics (116 papers)
- SGD noise and flat minima (112 papers)
- Stability-based generalization analysis (88 papers)
- Explicit and implicit regularization interplay (63 papers)
- Implicit regularization of SGD (55 papers)
- Implicit bias in linear networks (53 papers)
- Sharpness-aware minimization theory (37 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).