Research map / Learning theory and algorithms / Kernels and regression theory
High-dimensional asymptotic generalization
Derive exact asymptotic error of models in proportional high-dimensional limits using random matrix tools.
82 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Kernels and regression theory, keyword cluster “ridge · linear regression · regularization”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 19 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 8.5% to 7.7%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · The Teaching Dimension of Linear Learners (ICML 2016 · 19 citations)
- 2017 · Hypothesis Transfer Learning via Transformation Functions (NeurIPS 2017 · 36 citations)
- 2018 · Learning To Learn Around A Common Mean (NeurIPS 2018 · 39 citations)
- 2019 · On the Spectral Bias of Neural Networks (ICML 2019 · 151 citations)
- 2020 · Generalisation error in learning with random features and the hidden manifold model (ICML 2020 · 100 citations)
- 2021 · Provable Meta-Learning of Linear Representations (ICML 2021 · 49 citations)
- 2022 · Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension (ICML 2022 · 6 citations)
- 2023 · What learning algorithm is in-context learning? Investigations with linear models (ICLR 2023 · 86 citations)
- 2024 · A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · 7 citations)
- 2025 · Random Feature Representation Boosting (ICML 2025 · 0 citations)
- 2026 · Unifying Low Dimensional Spectra in Deep Learning (ICML 2026 · 2 citations)
Earliest papers here
- The Teaching Dimension of Linear Learners (ICML 2016 · 19 citations)
- Greedy Feature Construction (NeurIPS 2016 · 8 citations)
- On Regularizing Rademacher Observation Losses (NeurIPS 2016 · 1 citation)
What has been tried
59 papers here report 454 measured changes (a component added, removed or swapped, compared with a baseline): helped 58%, no clear effect 15%, hurt 18%, mixed 8%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use the approximate optimal weight matrices instead of the exact optimal matrices | Continual linear regression · estimation error | \gL(\widehat{\vw}_t^{(\mathrm{GR})}) \leq \frac{C}{e_j^{(0)} + \gamma_j^{(1)}n_1 + \dots + \gamma_j^{(t)} n_t}. | No clear effect | A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · 7 citations) |
| Use identity initialization instead of zero initialization | Airfoil, k=2, p=5 · R2 | 0.66±0.02 -> 0.87±0.01 | Helped | TMPNN: High-Order Polynomial Regression Based on Taylor Map Factorization (AAAI 2024 · 3 citations) |
| Increase ambient dimension d | Fast-decaying covariance spectra (λᵢ ∼ 2⁻ⁱ and λᵢ ∼ i⁻²) · ICL performance | No clear effect | How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? (ICLR 2024 · 1 citation) | |
| use gradient-greedy rather than exact-greedy RFRBoost | OpenML regression · RMSE | 0.408 -> 0.408 | No clear effect | Random Feature Representation Boosting (ICML 2025 · 0 citations) |
| Use Maximum Distance with repetition instead of single-pass Maximum Distance | Random data · average loss | Helped | Are Greedy Task Orderings Better Than Random in Continual Linear Regression? (NeurIPS 2025 · 0 citations) | |
| use the invariant operator Δ instead of sum aggregation | O(3) Action recognition · accuracy | 69.86 -> 87.36 | Helped | O$n$ Learning Deep O($n$)-Equivariant Hyperspheres (ICML 2024 · 0 citations) |
What reviewers will expect
Datasets and baselines named in the 10 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 2 of 10 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 40 papers; from OpenAlex and the papers' affiliations).
- Swiss Federal Institute of Technology Lausanne · 4 papers
- University of California, Berkeley · 3 papers
- Swiss Federal Institute of Technology · 3 papers
- Meta · 2 papers
- University of California, Los Angeles · 2 papers
- ETHZ - ETH Zurich · 2 papers
- Ecole Normale Supérieure · 2 papers
- Peking University · 2 papers
- Technion - Israel Institute of Technology · 2 papers
- EPFL · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- High-dimensional linear regression theory (61 papers)
- Double descent and overfitting (53 papers)
- Kernel ridge regression analysis (57 papers)
Elsewhere in the topic
- RKHS kernel methods (127 papers)
- Neural tangent kernel analysis (109 papers)
- Statistical perspectives on generalization (99 papers)
- Information-theoretic generalization bounds (94 papers)
- Kernel regression under shift (89 papers)
- Generalization theory of deep networks (67 papers)
- Empirical generalization measures (61 papers)
- PAC-Bayes bounds (39 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).