Research map / Learning theory and algorithms / Kernels and regression theory
Pooling Versus Ensembling for Ridge Regression Under Covariate Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares pooled ridge/ridgeless regression against optimally weighted ensembles of per-cluster learners under covariate-mean shift and a shared linear outcome model, deriving asymptotic risk formulas in the proportional regime.
Problem
Whether cluster-aware partitioning and ensembling of per-cluster ridge learners beats pooling when clusters differ only in covariate distribution (generalized covariate shift, no concept shift).
Approach
Derives exact asymptotic risk formulas for pooled and ensembled ridge and ridgeless predictors under mean-shifted covariates and linear outcome, showing under random effects an optimally tuned pooled ridge always beats optimally tuned, optimally weighted ensembles, with ridgeless exceptions near the interpolation threshold.
Key results
- Under random effects, the optimally tuned pooled ridge predictor has asymptotically no larger risk than the optimally tuned, optimally weighted ensemble (ratio of ensemble to pooled excess risk exceeds 1 for K=2).
- For ridgeless K=2 with equal cluster sizes and gamma<1/2, the ensemble-to-pooled risk ratio tends to (1-gamma)/(1-2gamma) > 1.
- Simulations show ridge MSE ratio between 1.06 and 1.62 across p, with no regime favoring the ensemble.
Surprising findings
- Randomly splitting data into halves (ignoring true clusters) gives identical ensemble-vs-pooled results, suggesting the aspect ratio rather than cluster structure drives the gap.
- For ridgeless regression, the ensemble can beat pooling when the pooled learner is at its interpolation threshold and components are overparameterized, due to implicit regularization.
Limitations
- Well-specified linear model with shared coefficients and noise variance; no misspecification or concept shift.
- Cluster labels assumed known; estimated clusters not analyzed.
- Fixed-effects optimal ensemble weights not derived in general.
Experimental setup
Datasets: synthetic Gaussian covariates with cluster mean shifts
Benchmarks: synthetic simulations
Metrics: MSE ratio of ensemble to pooled asymptotic prediction risk
Baselines: pooled ridge regression ensemble of per-cluster ridge learners ridgeless least squares
Builds on: asymptotic risk of ridge and ridgeless regression subagging and splagging analysis divide-and-conquer kernel ridge regression
Closest papers by meaning
- Fairness under Covariate Shift: Improving Fairness-Accuracy Tradeoff with Few Unlabeled Test Samples (AAAI 2024 · similarity 0.86)
- On Efficient Estimation of Distributional Treatment Effects under Covariate-Adaptive Randomization (ICML 2025 · similarity 0.85)
- An Effective Theory of Bias Amplification (ICLR 2025 · similarity 0.84)
- Hierarchical and Density-based Causal Clustering (NeurIPS 2024 · similarity 0.84)
- TIC-TAC: A Framework For Improved Covariance Estimation In Deep Heteroscedastic Regression (ICML 2024 · similarity 0.84)
- Asymptotically Free Sketched Ridge Ensembles: Risks, Cross-Validation, and Tuning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.