Research map / Learning theory and algorithms
Kernels and regression theory: research map
1,272 papers on Kernels and regression theory in Learning theory and algorithms, from ICML, NeurIPS, ICLR, CVPR, AAAI, ICCV and ECCV, and the journals TPAMI, IJCV and JMLR (2016–2026), grouped into 3 clusters and 14 approaches. The busiest year so far is 2024.
What it is
Theory of why models generalise from training to test data: generalisation bounds, kernel methods, and exact high-dimensional analyses of regression that explain effects such as double descent.
Terms you will see on this page:
- Generalization bound
- a guarantee that test error is close to training error.
- Kernel method
- learning with a similarity function that implicitly maps inputs to a rich feature space.
- Double descent
- test error falling again after the point where the model can fit the training data exactly.
Start reading with the most cited papers here:
- Exploring Generalization in Deep Learning (NeurIPS 2017 · 450 citations)
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (ICLR 2018 · 296 citations)
Within Learning theory and algorithms, its share held steady from 11.1% in 2023–24 to 10.2% in 2025–26 (248 → 247 papers at ICML, NeurIPS, ICLR, CVPR and AAAI, the venues with data for all four years).
Explore Kernels and regression theory in the interactive map
Working on something in this topic? Describe your idea in scime atlas to see which approach it falls under, the closest papers by meaning and how crowded the spot has become.
What changed recently
Approaches whose share of the topic moved most between 2023–24 and 2025–26 (ICML, NeurIPS, ICLR, CVPR and AAAI):
- Kernel two-sample testing rising: 1% → 5% of the topic (2 → 13 papers)
- Kernel regression under shift rising: 8% → 10% of the topic (19 → 25 papers)
- Information-theoretic generalization bounds rising: 9% → 11% of the topic (22 → 27 papers)
- Double descent and overfitting falling: 8% → 3% of the topic (19 → 7 papers)
- Generalization theory of deep networks falling: 8% → 5% of the topic (19 → 12 papers)
Papers from 2025–2026 that are already cited most:
- Error Bounds for Gaussian Process Regression Under Bounded Support Noise with Applications to Safety Certification (AAAI 2025 · 5 citations)
- Generalization Analysis for Deep Contrastive Representation Learning (AAAI 2025 · 1 citations)
- DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testing (NeurIPS 2025 · 0 citations)
- Kernel conditional tests from learning-theoretic bounds (NeurIPS 2025 · 0 citations)
Approaches and key papers
Kernel methods · 540 papers
Keywords: kernels, hilbert, reproducing
Approaches in this cluster:
- RKHS kernel methods (185 papers)
Analyze and accelerate kernel methods with Hilbert space theory, Nystrom and kernel thinning. - Neural tangent kernel analysis (122 papers)
Study wide networks via the neural tangent kernel and feature-learning limits. - Kernel regression under shift (141 papers)
Provide consistency and learning guarantees for kernel estimators in varied settings. - Kernel two-sample testing (47 papers)
Learn and combine kernels for MMD-based two-sample and independence tests. - Random feature approximation (45 papers)
Approximate kernels with random Fourier and orthogonal features, with error bounds.
Most cited and most cited since 2024:
- Exponential expressivity in deep neural networks through transient chaos (NeurIPS 2016 · 290 citations)
- Object Tracking via Dual Linear Structured SVM and Explicit Feature Map (CVPR 2016 · 264 citations)
- PINNACLE: PINN Adaptive ColLocation and Experimental points selection (ICLR 2024 · 6 citations)
- Error Bounds for Gaussian Process Regression Under Bounded Support Noise with Applications to Safety Certification (AAAI 2025 · 5 citations)
Generalization bounds · 401 papers
Keywords: generalization bounds, pac, generalization error
Approaches in this cluster:
- Statistical perspectives on generalization (109 papers)
Analyze generalization via invariance, interpolation, shift and information-theoretic views of data. - Information-theoretic generalization bounds (103 papers)
Bound generalization error with mutual information and conditional mutual information. - Generalization theory of deep networks (78 papers)
Study generalization of ReLU, ResNet and deep architectures via width, depth and sparsity. - Empirical generalization measures (66 papers)
Measure and predict generalization of deep networks through experiments and scaling laws. - PAC-Bayes bounds (45 papers)
Derive PAC-Bayesian generalization bounds with data-dependent priors and for meta-learning.
Most cited and most cited since 2024:
- Exploring Generalization in Deep Learning (NeurIPS 2017 · 450 citations)
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (ICLR 2018 · 296 citations)
- Improving Neural Network Generalization on Data-Limited Regression with Doubly-Robust Boosting (AAAI 2024 · 12 citations)
- A Unified Recipe for Deriving (Time-Uniform) PAC-Bayes Bounds (NeurIPS 2024 · 6 citations)
High-dimensional regression · 331 papers
Keywords: ridge, linear regression, regularization
Approaches in this cluster:
- High-dimensional asymptotic generalization (99 papers)
Derive exact asymptotic error of models in proportional high-dimensional limits using random matrix tools. - High-dimensional linear regression theory (106 papers)
Analyze risk of least-squares, interpolators and SVR in high dimensions. - Double descent and overfitting (58 papers)
Explain benign and harmful overfitting and double descent in overparameterized models. - Kernel ridge regression analysis (68 papers)
Characterize learning curves and test error of kernel ridge regression.
Most cited and most cited since 2024:
- Generalization Properties of Learning with Random Features (NeurIPS 2017 · 184 citations)
- On the Spectral Bias of Neural Networks (ICML 2019 · 151 citations)
- A Statistical Theory of Regularization-Based Continual Learning (ICML 2024 · 7 citations)
- Model Collapse Demystified: The Case of Regression (NeurIPS 2024 · 6 citations)
Related topics in Learning theory and algorithms
- Matrix and tensor methods (1,467)
- Sample complexity (1,553)
- Combinatorial and search optimization (1,576)
- Fairness (550)
- Submodular and game algorithms (1,064)
- Prediction and decision losses (2,400)
- Clustering (998)
- Optimal transport (462)
