Research map / Label-efficient and robust learning / Recognition and long-tail
Features are fate: a theory of transfer learning in high-dimensional regression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study of transfer learning (linear transfer and fine-tuning) in deep linear networks and student-teacher ReLU networks, showing that transferability is determined by the pretrained feature space overlap with the target task rather than distributional distance.
Problem
When does transfer learning from a pretrained model outperform scratch training on a data-limited target task, and what determines task similarity for transfer?
Approach
Adopts a feature-centric view: in the feature learning regime, pretraining sparsifies hidden features onto the source function, so transferability is governed by the source-target feature overlap angle and target sample size, yielding an exact phase diagram for linear transfer and fine-tuning in deep linear networks.
Key results
- Linear transfer generalization error derived as sin^2(theta) + (sigma^2 + sin^2(theta))/(n-2), so negative transfer occurs in regions of target size and task overlap.
- Fine-tuning transferability in deep linear networks is (gamma-1)(1-2cos(theta)) for gamma<=1 and zero for gamma>1, so fine-tuning is positive only when cos(theta)>1/2 in the overparameterized regime.
- Ridge regularization on linear transfer increases the generalization error for any lambda>0, enlarging the negative transfer region.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use linear transfer instead of scratch training | Deep linear network · Gaussian linear regression, γ<1 | expected generalization error: \frac{(1-\gamma)^2 + \gamma \sigma^2}{1-\gamma} - \sin^2\theta | Mixed |
| Use linear transfer instead of scratch training | Deep linear network · Gaussian linear regression, γ>1 | expected generalization error: \frac{\sigma^2}{\gamma -1} - \sin^2\theta | Mixed |
| Fine-tune all parameters from the pretrained initialization | Deep linear network · Gaussian linear regression, γ≤1 | transferability: (\gamma - 1)(1-2\cos\theta) | Mixed |
| Fine-tune all parameters from the pretrained initialization | Deep linear network · Gaussian linear regression, γ>1 | transferability: 0 | No clear effect |
| Add an ℓ2 penalty to linear transfer (ablation) | Deep linear network · Gaussian linear regression | expected generalization error: 1 - \frac{(1+ 2\lambda)}{(1+\lambda)^2}\cos^2\theta | Hurt |
| Increase source-target feature discrepancy μ (ablation) | Linear transfer · Nonlinear student-teacher ReLU regression | irreducible generalization error | Hurt |
| Train the source model in the rich rather than lazy feature-learning regime (ablation) | Pretrained nonlinear model · Transfer learning | negative transfer | Mixed |
| Use a finite source dataset instead of population pretraining (ablation) | Fine-tuning · Gaussian linear regression | transferability: \mathcal{T}_{\mathrm{ft}} = \begin{cases} \frac{\gamma_t-1}{1-\gamma_s}\sigma_s^2 \gamma_s + \gamma_s(\gamma_t-1)(1 - 2\ | Mixed |
Surprising findings
- Anomalous positive transfer occurs even when source and target feature spaces are orthogonal (theta = pi/2), driven only by the large source dataset.
- Distributional distances (KL, Wasserstein-1, Dudley metric) need not correlate with transfer efficiency; arbitrarily distant distributions within the same feature space can still transfer positively.
Limitations
- Analysis restricted to deep linear networks, which are limited to affine functions and independent of depth.
- Nonlinear scratch-trained generalization is approximated by an empirical power law and the transfer boundary is heuristic.
- Scratch-trained baseline is unregularized, so anomalous positive transfer must be interpreted with care.
Experimental setup
Datasets: synthetic Gaussian linear regression data synthetic student-teacher ReLU data
Benchmarks: synthetic linear regression synthetic ReLU student-teacher regression
Metrics: generalization error transferability
Baselines: scratch training ridge-regularized linear transfer
Builds on: Yun et al. implicit bias of deep linear networks neural tangent kernel mean field neural network theory
Closest papers by meaning
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.87)
- On Transferring Transferability: Towards a Theory for Size Generalization (NeurIPS 2025 · similarity 0.87)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
- Transferring Learning Trajectories of Neural Networks (ICLR 2024 · similarity 0.86)
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.86)
- On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.