Research map / Learning theory and algorithms / Kernels and regression theory
The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift
Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, Sham M. Kakade
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add target-data finetuning after source pretraining | Online SGD for linear regression under covariate shift · Instances where target and source covariance matrices are suitably aligned | source data required to match supervised target risk: finetuning can reduce by at least constant factors the amount of source data required by pretraining | Improved |
| Combine source pretraining with a small amount of target finetuning | Online SGD for linear regression under covariate shift · Example with covariance shift parameterized by ε | total samples sufficient for excess risk below ε: M \gtrsim \epsilon^{-2}; N \gtrsim \epsilon^{-1.5}; M \eqsim \epsilon^{-1} \log(\epsilon^{-1}), $N \eqsim \epsilon^{-1}\ | Improved |
| Use source-only pretraining without finetuning | Online SGD for linear regression under covariate shift · Synthetic P(k = 5), strong covariance alignment | target excess risk | No clear effect |
| Add scarce target-data finetuning to pretraining | Online SGD for linear regression under covariate shift · Synthetic P(k = 10), moderate covariance alignment | target excess risk | Improved |
| Add finetuning to pretraining under severe covariance misalignment | Online SGD for linear regression under covariate shift · Synthetic P(k = 20), poor covariance alignment | target samples needed to match supervised risk: nearly the same amount of target data as that used by supervised learning | No clear effect |
| Use pretraining-finetuning rather than pretraining alone when target data are available | Online SGD for linear regression under covariate shift · Any instance, with optimally tuned stepsizes | target excess risk | No clear effect |
| Apply the derived analysis to target-only last-iterate SGD | Supervised linear regression · Target-only supervised learning | excess-risk upper bound: by a logarithmic factor | Improved |
Closest papers by meaning
- Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.88)
- Minimum-Norm Interpolation Under Covariate Shift (ICML 2024 · similarity 0.87)
- Transfer Learning Meets Functional Linear Regression: No Negative Transfer Under Posterior Drift (AAAI 2025 · similarity 0.85)
- Computational Efficiency under Covariate Shift in Kernel Ridge Regression (NeurIPS 2025 · similarity 0.85)
- Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression (NeurIPS 2025 · similarity 0.85)
- Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
