Research map / Label-efficient and robust learning / Recognition and long-tail
On the Provable Advantage of Unsupervised Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a generic theory showing MLE-based unsupervised pretraining followed by ERM on labeled data achieves excess risk Õ(sqrt(C_Φ/m)+sqrt(C_Ψ/n)) under an informative condition, instantiated for factor models, Gaussian mixtures, and contrastive learning.
Problem
Explaining provably when unsupervised pretraining reduces sample complexity of downstream supervised tasks, beyond specific pretraining methods with restrictive structural assumptions.
Approach
An abstract latent-variable framework: MLE on abundant unlabeled data learns the latent model, and a "kappa-informative" condition (allowing transformations induced by the downstream task) ensures this yields a representation that bounds downstream excess risk, so the labeled-data complexity term depends only on the predictor class.
Key results
- MLE+ERM achieves excess risk Õ(sqrt(C_Φ/m)+sqrt(C_Ψ/n)) under the kappa^{-1}-informative condition, versus Õ(sqrt(C_{Φ∘Ψ}/n)) for supervised-only learning.
- Factor model with linear regression yields fast-rate excess risk O(d log(1/δ)/m + r log(4/δ)/n) under sufficient sample sizes.
- GMM with classification gives Õ(sqrt(dK/m)+sqrt(K/n)) when m = Ω̃(dK^3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unlabeled MLE pretraining before labeled ERM instead of supervised learning on labeled data alone | Generic downstream tasks under the informative condition · Generic downstream tasks | excess risk: \tilde{\mathcal{O}}(\sqrt{\mathcal{C}_\Phi/m} + \sqrt{\mathcal{C}_\Psi/n}) versus \tilde{\mathcal{O}}(\sqrt{\mathcal{C}_ | Helped |
| Add unlabeled factor-model pretraining | Linear regression on factor-model data · Factor models with linear regression | excess risk: \Tilde{\mathcal{O}}(d/m + r/n) versus \Tilde{\mathcal{O}}(d/n) | Helped |
| Add unlabeled GMM pretraining | Classification with Gaussian mixture model data · Gaussian mixture model classification | excess risk: \tilde{\mathcal{O}}(\sqrt{dK/m}+\sqrt{K/n}) versus \tilde{\mathcal{O}}(\sqrt{dK/n}) | Helped |
| Add contrastive pretraining | Linear regression on contrastive representations · Contrastive learning with linear regression | excess risk: \tilde{O}(\sqrt{1/n}) versus direct supervised learning, whose excess risk strongly depends on the complexity of \mathca | Helped |
| Remove the informative condition | Unsupervised pretraining for generic latent-variable models · Generic downstream tasks | additional information for learning the predictor | Hurt |
| Use MLE rather than ERM in the downstream phase (ablation) | Two-phase MLE+ERM · Generic downstream tasks | worst-case downstream performance | Hurt |
Surprising findings
- The two-phase MLE scheme (MLE for both phases) can fail in the worst case, unlike MLE followed by ERM.
Limitations
- Results are upper bounds in abstract function classes; the informative condition is assumed and may be strong.
- Factor-model rate is not optimal under the generic analysis and needs a specialized fast-rate argument.
- No empirical experiments validating the bounds.
Experimental setup
Metrics: excess risk total variation distance Hellinger distance
Baselines: supervised learning on labeled data only (ERM baseline) two-phase MLE scheme
Builds on: maximum likelihood estimation analysis (bracketing) Rademacher complexity bounds contrastive learning theory
Closest papers by meaning
- Unsupervised Learning via Meta-Learning (ICLR 2019 · similarity 0.89)
- Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Models (ICML 2025 · similarity 0.88)
- Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing (NeurIPS 2026 · similarity 0.87)
- Machine Unlearning under Overparameterization (NeurIPS 2025 · similarity 0.86)
- When to Forget? Complexity Trade-offs in Machine Unlearning (ICML 2025 · similarity 0.85)
- How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.