Research map / Optimization / Implicit bias and generalization
Universality in Transfer Learning for Linear Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves Gaussian universality for regularized least squares and SMD/SGD implicit-regularization solutions under block-regular non-Gaussian designs, and applies it to exact fine-tuning error analysis for linear regression and binary classification transfer learning.
Problem
Characterizing the exact generalization and classification error of linear models fine-tuned by SGD from pretrained weights on small target data, beyond Gaussian design assumptions, in the overparameterized proportional regime.
Approach
Establishes that test and training errors of regularized least-squares and SMD/SGD interpolating solutions depend only on the first and second moments of the design rows (Gaussian universality via Lindeberg-type replacement), allowing exact transfer-learning formulas for non-Gaussian data, including mixture classification with arbitrary class covariances.
Key results
- Regression: fine-tuned generalization error e_p is lower-bounded by sigma^2/(kappa-1) + (kappa-1)/kappa * e_a, with equality iff the covariance is scalar.
- Classification: closed-form error prediction e_p = Q(...) matches SGD simulations across normal, Bernoulli, and chi-squared entries.
- Fine-tuning helps only when the signal-to-noise parameter rho is Theta(1) or larger; for rho<1 transfer learning fails for large kappa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the pretrained regressor on target data | Linear regression with SGD · Linear regression | generalization error: e_p \ge \frac{1}{\kappa-1} \sigma^2 + \frac{\kappa-1}{\kappa} e_a | Mixed |
| Fine-tune the pretrained regressor when noise variance is at least the prior error | Linear regression with SGD · Linear regression, σ² ≥ e_a | generalization error: e_p \ge \frac{1}{\kappa-1} \sigma^2 + \frac{\kappa-1}{\kappa} e_a \ge e_a | Hurt |
| Fine-tune with scalar data covariance and noise variance below the prior error | Linear regression with SGD · Linear regression, scalar covariance and e_a ≥ σ² | generalization error: best achievable error ... equals σ (2√e_a-σ), which is less than e_a | Helped |
| Fine-tune the pretrained classifier | Binary classification with SGD · Binary classification, ρ = ω(1) | classification error: e_p(κ, ∞)\xrightarrow[]{\mathbb{P}} Q\biggl(\Bigl(Q^{-1}(e_a)+\frac{1}{Q^{-1}(e_a)}\Bigr)\sqrt{\frac{κ}{κ-1}}\biggr) < e | Helped |
| Fine-tune the pretrained classifier | Binary classification with SGD · Binary classification, ρ = Θ(1) | classification error: e_p(∞, ρ)\xrightarrow[]{\mathbb{P}} e_a | Mixed |
| Fine-tune the pretrained classifier | Binary classification with SGD · Binary classification, ρ = o(1) | classification error: Q(\sqrt{\frac{ρ}{2}}) | No clear effect |
| Fine-tune the pretrained classifier when ρ < 1 and κ ≫ 1 | Binary classification with SGD · Binary classification, highly overparameterized regime | classification error | Hurt |
| Fine-tune the pretrained classifier for sufficiently poor initialization when ρ > 1 and κ is large enough | Binary classification with SGD · Binary classification, highly overparameterized regime | classification error: e_a > Q(\sqrt{\frac{ρ-1}{3}}) > Q(\sqrt{\frac{ρ}{2}}) | Helped |
Surprising findings
- If noise variance sigma^2 >= e_a, fine-tuning is provably no better than using the pretrained vector in regression.
- Fine-tuning can hurt performance for small kappa but help once kappa is large enough in some classification settings.
Limitations
- Results hold asymptotically in the proportional regime d/n to kappa, with d=Theta(n).
- Implicit-regularization analysis requires extra regularity; the classification application depends on a quadratic regularizer and Gaussian-equivalence CLT verified case by case.
- Experiments are on synthetic data only, with d=1000 for regression.
Experimental setup
Datasets: synthetic Gaussian, Bernoulli, and chi-squared designs
Metrics: generalization error classification error
Baselines: Gaussian-design transfer learning analysis pretrained weights without fine-tuning
Builds on: stochastic mirror descent Gaussian equivalence principle Lindeberg replacement method
Closest papers by meaning
- Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.86)
- Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning (NeurIPS 2025 · similarity 0.86)
- Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models (ICML 2025 · similarity 0.86)
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (ICLR 2024 · similarity 0.86)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.85)
- Features are fate: a theory of transfer learning in high-dimensional regression (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.