AI Paper Atlas

Research map / Learning theory and algorithms / Kernels and regression theory

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift

Jingfeng Wu, Difan Zou, Vladimir Braverman, Quanquan Gu, Sham M. Kakade

NeurIPS 2022 · Learning theory and algorithms › Kernels and regression theory · approach: High-dimensional linear regression theory · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Add target-data finetuning after source pretrainingOnline SGD for linear regression under covariate shift · Instances where target and source covariance matrices are suitably alignedsource data required to match supervised target risk: finetuning can reduce by at least constant factors the amount of source data required by pretrainingImproved
Combine source pretraining with a small amount of target finetuningOnline SGD for linear regression under covariate shift · Example with covariance shift parameterized by εtotal samples sufficient for excess risk below ε: M \gtrsim \epsilon^{-2}; N \gtrsim \epsilon^{-1.5}; M \eqsim \epsilon^{-1} \log(\epsilon^{-1}), $N \eqsim \epsilon^{-1}\Improved
Use source-only pretraining without finetuningOnline SGD for linear regression under covariate shift · Synthetic P(k = 5), strong covariance alignmenttarget excess riskNo clear effect
Add scarce target-data finetuning to pretrainingOnline SGD for linear regression under covariate shift · Synthetic P(k = 10), moderate covariance alignmenttarget excess riskImproved
Add finetuning to pretraining under severe covariance misalignmentOnline SGD for linear regression under covariate shift · Synthetic P(k = 20), poor covariance alignmenttarget samples needed to match supervised risk: nearly the same amount of target data as that used by supervised learningNo clear effect
Use pretraining-finetuning rather than pretraining alone when target data are availableOnline SGD for linear regression under covariate shift · Any instance, with optimally tuned stepsizestarget excess riskNo clear effect
Apply the derived analysis to target-only last-iterate SGDSupervised linear regression · Target-only supervised learningexcess-risk upper bound: by a logarithmic factorImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy