Research map / Learning theory and algorithms / Kernels and regression theory
Information-theoretic generalization bounds
Bound generalization error with mutual information and conditional mutual information.
94 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Kernels and regression theory, keyword cluster “generalization bounds · pac · generalization error”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
22 papers in 2023–24 and 27 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 8.9% to 10.9%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Information-theoretic analysis of generalization capability of learning algorithms (NeurIPS 2017 · 199 citations)
- 2018 · Multi-Class Learning: From Theory to Algorithm (NeurIPS 2018 · 34 citations)
- 2019 · Uniform Error Bounds for Gaussian Process Regression with Application to Safe Control (NeurIPS 2019 · 52 citations)
- 2020 · Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms (NeurIPS 2020 · 48 citations)
- 2021 · Tighter Expected Generalization Error Bounds via Wasserstein Distance (NeurIPS 2021 · 24 citations)
- 2022 · A New Family of Generalization Bounds Using Samplewise Evaluated CMI (NeurIPS 2022 · 5 citations)
- 2023 · StableDR: Stabilized Doubly Robust Learning for Recommendation on Data Missing Not at Random (ICLR 2023 · 12 citations)
- 2024 · Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting (AAAI 2024 · 12 citations)
- 2025 · Provable Length Generalization in Sequence Prediction via Spectral Filtering (ICML 2025 · 0 citations)
- 2026 · Harmful Overfitting in Sobolev Spaces (ICML 2026 · 0 citations)
Earliest papers here
- Information-theoretic analysis of generalization capability of learning algorithms (NeurIPS 2017 · 199 citations)
- Nearest-Neighbor Sample Compression: Efficiency, Consistency, Infinite Dimensions (NeurIPS 2017 · 5 citations)
- Generalization Analysis for Ranking Using Integral Operator (AAAI 2017 · 4 citations)
What has been tried
58 papers here report 368 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 3%, hurt 17%, mixed 7%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add the PLSR statistical learner | ELECTRICITY · R2 | 0.836±0.009 -> 0.844±0.007 | Helped | Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting (AAAI 2024 · 12 citations) |
| Add a second autoregressive component, y_{t-2} | Synthetic LDS, eigenvalues sampled from Region B · prediction loss | \sqrt{T} | Helped | Provable Length Generalization in Sequence Prediction via Spectral Filtering (ICML 2025 · 0 citations) |
| Vary the variational ansatz from QCNN to QCNN with shared parameters or Strongly Entangling Layers | QPR · generalization-gap prediction by margin metrics | Helped | Understanding Generalization in Quantum Machine Learning with Margins (ICML 2025 · 0 citations) | |
| Decrease the Lipschitz constant | Interval-target experiments · average interval size | Helped | Learning from Interval Targets (NeurIPS 2025 · 0 citations) | |
| Train on a lower-dimensional random subspace | Gaussian mean estimation · generalization error | Helped | Slicing Mutual Information Generalization Bounds for Neural Networks (ICML 2024 · 0 citations) | |
| Use the lossy rather than the lossless Gaussian mixture prior regularizer | Experiments across datasets · population accuracy | Helped | Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025) |
What reviewers will expect
Datasets and baselines named in the 19 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MNIST · 3 of 19 papers
- CIFAR10 · 2 of 19 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 49 papers; from OpenAlex and the papers' affiliations).
- University of Ottawa · 3 papers
- Xi’an Jiaotong University · 2 papers
- RIKEN · 2 papers
- University of Haifa · 2 papers
- EPFL · 2 papers
- Meta · 2 papers
- Imperial College London · 2 papers
- Tsinghua University · 2 papers
- Université Gustave Eiffel · 2 papers
- Huawei · 2 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Statistical perspectives on generalization (99 papers)
- Generalization theory of deep networks (67 papers)
- Empirical generalization measures (61 papers)
- PAC-Bayes bounds (39 papers)
Elsewhere in the topic
- RKHS kernel methods (127 papers)
- Neural tangent kernel analysis (109 papers)
- Kernel regression under shift (89 papers)
- High-dimensional asymptotic generalization (82 papers)
- High-dimensional linear regression theory (61 papers)
- Kernel ridge regression analysis (57 papers)
- Double descent and overfitting (53 papers)
- Random feature approximation (36 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).