Research map / Optimization / Implicit bias and generalization
Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes how unsupervised PCA pre-training and transfer learning reduce online SGD sample complexity for single-index models with spiked-covariance Gaussian features, and shows cases where random initialization provably fails.
Problem
Sample complexity of online SGD for training a single-layer single-index network in high dimension, comparing random initialization against unsupervised pre-training (PCA on unlabeled features) and transfer learning from a correlated source vector.
Approach
Model unlabeled features as Gaussian with spiked covariance whose spike is correlated with the target, so PCA on unlabeled data yields an informative initialization; under concept shift, a correlated source vector initializes SGD. Uses population-flow analysis and martingale concentration to derive polynomial (and in some cases exponential) sample-complexity gaps versus random initialization.
Key results
- With PCA pre-training and correlation eta_1 >= m_1*, SGD with N=alpha*d (alpha=omega(1)) samples is effective, i.e. near-linear sample complexity.
- For random initialization with information exponent >=3 Hermite-type activations, m_1(X_N) -> 0 for N=alpha*d with alpha << d, implying Omega(d^2) samples are needed, giving a polynomial separation d^zeta for zeta<1.
- When eta_1=1, there exists r>0 such that any initialization within |m_1|<r fails to recover the target for any number of samples in the stated stepsize regime, an exponential-type separation from pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Initialize with a PCA estimate from unlabeled features instead of a uniform random vector | Spherical SGD for a spiked-covariance single-index model, under the stated basin and alignment conditions · Spiked-covariance single-index model | labeled sample complexity: N = αd, α = ω(1) versus N = Ω(d^2) at minimum | Helped |
| Initialize randomly rather than with a suitably aligned PCA estimate | Spherical SGD for the spiked-covariance single-index model · Spiked-covariance single-index model | labeled sample complexity: N = Ω(d^2) at minimum versus N = αd, α = ω(1) | Hurt |
| Use a transfer initialization with correlation η_d = θ(d^{-ζ}) instead of random initialization | Spherical SGD for an isotropic Gaussian single-index model with information exponent k · Isotropic Gaussian transfer learning | labeled sample complexity: α ≫ d^{k-2}(log d)^2 to α ≫ d^{2ζ(k-2)} (log d)^{2 1[ζ > 0]} | Helped |
| Initialize randomly rather than with fixed positive correlation | Spherical SGD for an isotropic Gaussian single-index model with information exponent k · Isotropic Gaussian transfer learning | labeled sample complexity: α ≫ d^{k-2}(log d)^2 versus α ≫ d^{2ζ(k-2)} (log d)^{2 1[ζ > 0]} | Hurt |
| Use random initialization when the spike is perfectly aligned with the target | Spherical SGD for a spiked-covariance single-index model with η₁ = 1 · Spiked-covariance single-index model, η₁ = 1 | recovery correlation m₁(X_N): m₁(X_N) → 0 | Hurt |
| Use a transfer initialization with fixed correlation (ζ = 0) | Spherical SGD for an isotropic Gaussian single-index model · Isotropic Gaussian transfer learning | labeled sample complexity: α = ω(1) | Helped |
Surprising findings
- With spiked covariance and eta_1=1, a local optimum at m_1=0 traps random and even fixed-correlation initializations, so SGD can lose initial information.
- Under transfer learning with fixed correlation (zeta=0), only alpha=omega(1) samples are needed regardless of information exponent.
Limitations
- Results are asymptotic in dimension and rely on specific Gaussian features, spherical SGD, and known activation f.
- Assumption on population dynamics (Assumption with point m*) must be verified per activation; general sufficiency only.
- Lower bound for random initialization only shows failure to recover with Omega(d^2) samples, not when recovery succeeds.
Experimental setup
Datasets: synthetic Gaussian single-index data with spiked covariance
Metrics: correlation m_1(X) with target after SGD convergence to (1,0) in probability
Baselines: random initialization
Builds on: information exponent (Ben Arous et al.) bounding flows spherical SGD analysis of single-index models
Closest papers by meaning
- On the Provable Advantage of Unsupervised Pretraining (ICLR 2024 · similarity 0.88)
- Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.86)
- Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning (NeurIPS 2025 · similarity 0.85)
- Online Learning and Information Exponents: The Importance of Batch size & Time/Complexity Tradeoffs (ICML 2024 · similarity 0.85)
- The Impact of Geometric Complexity on Neural Collapse in Transfer Learning (NeurIPS 2024 · similarity 0.85)
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.