Research map / Label-efficient and robust learning / Recognition and long-tail
Mixed-Sample SGD: an End-to-end Analysis of Supervised Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixed-Sample SGD, which alternates sampling between source and target data with an adaptive rate tracking a Lagrange multiplier, and proves it adaptively achieves transfer rates for convex losses and linear regression.
Problem
Supervised transfer learning requires choosing how much to weight source versus target data without knowing source quality; existing approaches focus on statistical guarantees and are computationally expensive (cross-validation, projections).
Approach
An SGD procedure whose source/target sampling rate is set by a stochastic Lagrange multiplier that tracks a sequence of constrained convex programs (minimize source risk subject to target risk near the target ERM), avoiding projections and cross-validation, with analysis handling unbounded constraint sets and a parallel target-ERM estimate.
Key results
- Theory: target excess risk of the output is bounded by a constant times min{epsilon_Q, lambda_max(Sigma_P^{-1} Sigma_Q) epsilon_P + E_Q(theta_P*)}, with SGD convergence at rate 1/sqrt(T).
- On the School dataset, the method consistently outperforms source ERM, target ERM and HTL and adapts to the better of source and target ERM, with runtime lower than PSGD.
- On synthetic data with low-rank Sigma_Q and n_Q=50, the method remains adaptive while HTL loses adaptivity due to cross-validation with little target data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive mixed-sample SGD instead of source ERM or target ERM | Mixed-Sample SGD · Synthetic linear regression | target risk | Helped |
| use adaptive mixed-sample SGD instead of source ERM or target ERM | Mixed-Sample SGD · Synthetic linear regression with increasing transfer hardness | target risk | Helped |
| use adaptive mixed-sample SGD with low-rank target covariance | Mixed-Sample SGD · Synthetic linear regression, low-rank target covariance | target risk | Helped |
| use adaptive mixed-sample SGD instead of source ERM, target ERM, or HTL | Mixed-Sample SGD · School Dataset | MSE | Helped |
| use mixed-sample SGD instead of PSGD | Mixed-Sample SGD · School Dataset | runtime | Helped |
| use adaptive mixed-sample SGD instead of HTL | Mixed-Sample SGD · CIFAR-10 Dog vs Cat, target training size varied | target error | Helped |
| use adaptive mixed-sample SGD instead of source ERM or HTL | Mixed-Sample SGD · Berkeley Yearbook | MSE | Helped |
| use mixed-sample SGD instead of HTL | Mixed-Sample SGD · Berkeley Yearbook, source sample size below 1300 | runtime | Helped |
| use mixed-sample SGD instead of HTL | Mixed-Sample SGD · Berkeley Yearbook, source sample size 1300 | runtime | Hurt |
| use adaptive mixed-sample SGD instead of source ERM as target sample size increases | Mixed-Sample SGD · CIFAR-10 Dog vs Cat, source data uninformative | target error | Helped |
| use HTL instead of mixed-sample SGD | Mixed-Sample SGD · CIFAR-10 Dog vs Cat, source data uninformative | target error | Hurt |
| increase the number of abnormal source samples | Mixed-Sample SGD · Malware IoT | Type-I and Type-II error | Helped |
Surprising findings
- HTL with cross-validation can fail to select a good bias parameter when target validation data is scarce, performing close to target ERM.
Limitations
- Theory is for linear regression with square loss in the main text; general convex losses rely on bounded-loss and Rademacher assumptions and extra constants.
- Some key parameters (rho, lambda*, Lipschitz constants) depend on unknown population quantities and require warm-up estimates.
- Experiments are small-scale with linear models on fixed features.
Experimental setup
Datasets: synthetic Gaussian regression School Dataset CIFAR-10 dog vs cat Malware IoT network traffic Berkeley Yearbook
Benchmarks: School Dataset CIFAR-10 dog vs cat Malware IoT Berkeley Yearbook
Metrics: MSE target error Type-I error Type-II error runtime
Baselines: source ERM target ERM projected SGD hypothesis transfer learning (HTL)
Builds on: hypothesis transfer learning projected SGD Lagrangian primal-dual SGD
Closest papers by meaning
- A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning (NeurIPS 2025 · similarity 0.88)
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.86)
- Universality in Transfer Learning for Linear Models (NeurIPS 2024 · similarity 0.86)
- On the Convergence of Loss and Uncertainty-based Active Learning Algorithms (NeurIPS 2024 · similarity 0.86)
- Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression (NeurIPS 2025 · similarity 0.85)
- Wasserstein Transfer Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.