Research map / Learning theory and algorithms / Kernels and regression theory
Theoretical Limitations of Ensembles in the Age of Overparameterization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that infinite ensembles of overparameterized random feature regressors are pointwise equivalent to a single infinite-width model (exact when ridgeless, Lipschitz-approximate for small ridge), and that ensemble variance measures sensitivity to capacity rather than classical uncertainty.
Problem
Whether ensembles of overparameterized models (e.g., deep ensembles) offer generalization or uncertainty advantages over a single larger model, unlike classic underparameterized ensembles.
Approach
Using minimal distributional assumptions (subexponential whitened features, no Gaussianity), shows the average of ridgeless overparameterized random feature regressors equals the infinite-width kernel regressor pointwise, so ensembling adds no benefit beyond capacity; ensemble variance equals expected squared difference between finite- and infinite-width predictions.
Key results
- Theorem 3.2: infinite ensemble of overparameterized ReLU/RF ridgeless regressors is pointwise almost surely equal to the infinite-width single RF regressor, regardless of feature distribution.
- Theorem 3.5: difference between ridge ensembles and single infinite-width model is Lipschitz in the ridge parameter, so |difference| <= C*lambda.
- Empirically, RF ensembles and single models with equal total feature budgets show nearly identical generalization error on California Housing (N=12, D=200).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Ensemble overparameterized RF regressors instead of using a single infinite-width RF regressor | Ridgeless overparameterized RF regression · Fixed training dataset | prediction | No clear effect |
| Use an overparameterized RF ensemble rather than a single larger RF model with the same total parameter budget | Overparameterized RF regression · California Housing | generalization error | No clear effect |
| Use a deep ensemble rather than a single larger neural network with the same total parameter budget | Three-layer MLPs · California Housing | generalization error | No clear effect |
| Increase RF model width into the overparameterized regime | RF ensembles · California Housing | prediction difference from the ridgeless kernel regressor | Helped |
| Use ensemble predictive variance as a conventional uncertainty estimate with general, non-Gaussian features | Overparameterized RF ensembles · ReLU random features | uncertainty calibration | Hurt |
| Assume Gaussian random features when interpreting ensemble variance | Overparameterized RF ensembles · Gaussian feature distributions | uncertainty interpretation | Helped |
| Introduce a small positive ridge penalty instead of using ridgeless RF regressors | Infinite RF ensembles and infinite-width RF models · Small-ridge regime | prediction difference between ensemble and single model | Hurt |
Surprising findings
- Ensemble variance under non-Gaussian (ReLU) random features deviates substantially from the Gaussian process posterior variance, so it does not carry a standard Bayesian uncertainty interpretation.
- Classic underparameterized ensemble benefits (ridge-like regularization) disappear when D > N; width no longer controls implicit regularization.
Limitations
- Results are exact only for ridgeless models; the small-ridge result is an approximation with Lipschitz bound.
- Analysis is for random feature regressors, which only partially explain neural network behavior; neural network experiments are limited.
- Finite-parameter-budget equivalence (Theorem 3.3) holds under slightly stronger assumptions and is non-asymptotic with O(1/D) and O(sqrt(log(1/delta))) terms.
Experimental setup
Datasets: California Housing synthetic data
Benchmarks: California Housing
Metrics: generalization error absolute difference between ensemble and single-model predic predictive variance
Baselines: Gaussian process posterior variance single large model (kernel model) underparameterized RF ensembles
Builds on: Jacot et al. 2020 ensemble of ridge random feature models random feature models (Rahimi & Recht) deep ensembles (Lakshminarayanan et al. 2017) Gaussian process posterior variance
Closest papers by meaning
- No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions (ICML 2025 · similarity 0.90)
- United We Stand: Using Epoch-Wise Agreement of Ensembles to Combat Overfit (AAAI 2024 · similarity 0.88)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.87)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.87)
- Spurious Feature Diversification Improves Out-of-distribution Generalization (ICLR 2024 · similarity 0.86)
- Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.