Research map / Learning theory and algorithms / Kernels and regression theory
Ridge Boosting is Both Robust and Efficient
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a single kernel-ridge boosting step on an initial predictor yields a predictor that is multiaccurate over an RKHS unit ball of density ratios/Riesz representers and is semiparametrically efficient for every target in that class.
Problem
Plug-in estimators of linear functionals of a regression function trained on a source distribution suffer regularization bias under covariate shift, and existing robust (multiaccurate) and efficient (bias-corrected) estimators trade off against each other; the paper asks whether one predictor can be
Approach
One boosting step with kernel ridge regression on the residuals of any initial predictor produces a multiaccurate predictor; its plug-in estimate is numerically equivalent to an automatic debiased (Riesz-regression) estimator, so it attains the semiparametric efficiency bound simultaneously for all targets whose Riesz representers lie in the RKHS unit ball.
Key results
- In simulations, naive kernel ridge 95% confidence intervals undercover across three covariate shifts, while once-boosted ridge intervals achieve near-nominal coverage by n=300.
- In the retirement income application, the naive plug-in age profile declines by about $11k from ages 65-89, while boosted estimates are substantially flatter.
- Ridge boosting's plug-in estimate is shown to be numerically equivalent to a Riesz-regression augmented estimator, yielding semiparametric efficiency for all targets with representers in the RKHS unit ball.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply one kernel ridge boosting step to the initial kernel ridge predictor | Kernel Ridge · Average derivative simulation, three test distributions (μ ∈ {0, 1, -1}) | 95% CI empirical coverage | Helped |
| Use the unboosted kernel ridge predictor instead of adding a ridge boosting step | Kernel Ridge · Average derivative simulation, three test distributions (μ ∈ {0, 1, -1}) | 95% CI empirical coverage | Hurt |
| Apply ridge boosting rather than use the naive kernel ridge plug-in | Kernel Ridge · Retirement income age profile, ages 65–89 | Estimated income profile: steep decline of $11k | Mixed |
| Apply ridge boosting rather than use the naive kernel ridge plug-in | Kernel Ridge · Retirement income age profile, ages 65–89 | 95% CI width | Mixed |
Surprising findings
- Naive confidence intervals from the base ridge model under-cover, while the boosted model reaches valid coverage, so a single boosted predictor serves many targets.
Limitations
- Results hold only when target Riesz representers lie in an RKHS unit ball, which excludes shifts requiring fine-tuning of the representation.
- Hyperparameter tuning (e.g., lambda) needs further investigation for practical use.
- Random forest extension is sketched only for ATE/covariate-shift estimands.
Experimental setup
Datasets: 2018 American Community Survey via FolkTables
Benchmarks: simulated average derivative under covariate shift ACS retirement income age profile
Metrics: empirical coverage of 95% confidence intervals confidence interval width
Baselines: naive kernel ridge plug-in
Builds on: multiaccuracy automatic debiased machine learning kernel ridge regression targeted maximum likelihood estimation
Closest papers by meaning
- One-shot Distributed Ridge Regression in High Dimensions (ICML 2020 · similarity 0.86)
- Ridge Regression: Structure, Cross-Validation, and Sketching (ICLR 2020 · similarity 0.86)
- Optimal Ridge Regularization for Out-of-Distribution Prediction (ICML 2024 · similarity 0.86)
- Risk and cross validation in ridge regression with correlated samples (ICML 2025 · similarity 0.85)
- Asymptotically Free Sketched Ridge Ensembles: Risks, Cross-Validation, and Tuning (ICLR 2024 · similarity 0.84)
- An Agnostic View on the Cost of Overfitting in (Kernel) Ridge Regression (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.