Research map / Label-efficient and robust learning / Recognition and long-tail
Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-examines MAP transfer learning with source-informed priors (LearnedPriorIso and LearnedPriorLR) against standard initialization-only transfer across five image datasets, finding standard transfer much stronger than previously reported and gains varying by dataset.
Problem
Whether learned source-informed weight priors (Bayesian transfer learning) truly improve over standard initialization-only transfer learning when few labeled target examples are available, given reported gains that may be inflated by weak baselines.
Approach
Careful, equal-effort hyperparameter tuning and a side-by-side reimplementation show standard fine-tuning from a SimCLR ResNet-50 initialization performs far better than previously reported; isotropic learned-mean priors are competitive with low-rank covariance priors, and the proposed loss-landscape alignment mechanism is not consistently supported.
Key results
- On CIFAR-10 with n=1000, StdPrior reaches 80.9% mean accuracy (78.2-82.8 range) versus 64% previously reported for standard transfer learning.
- Informed priors give substantial gains only on FGVC-Aircraft (e.g., 51.3% LearnedPriorIso vs 40.6% StdPrior at n=1000), negligible on Oxford Flowers and HAM10000, and modest 1.5-3 points on CIFAR-10 and Oxford-IIIT Pets.
- LearnedPriorIso is competitive with LearnedPriorLR across datasets while being simpler to tune; loss-landscape alignment varies considerably across replicate training sets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an informative prior rather than initialization-only transfer learning | CIFAR-10 transfer learning · CIFAR-10, 10-100 examples per class | accuracy: 58.7% for LearnedPriorLR vs. 56.7% for LearnedPriorIso | Helped |
| Use an informative prior rather than initialization-only transfer learning | Oxford Flowers transfer learning · Oxford Flowers, 5 examples per class | accuracy: 78.9; 79.1; 77.9 | No clear effect |
| Use an informative prior rather than initialization-only transfer learning | Oxford-IIIT Pets transfer learning · Oxford-IIIT Pets, 10 examples per class | accuracy: 57.4% accuracy for LearnedPriorLR vs. 55.5% accuracy for LearnedPriorIso | Helped |
| Use an informative prior rather than initialization-only transfer learning | FGVC-Aircraft transfer learning · FGVC-Aircraft, 5-10 examples per class | accuracy: 25.9; 24.5; 22.5; 51.3; 50.9; 40.6 | Helped |
| Use an informative prior rather than initialization-only transfer learning | HAM10000 transfer learning · HAM10000, n=100-1000 | average AUROC: 78.1; 78.0; 78.7; 85.6; 86.5; 86.6 | No clear effect |
| Use low-rank covariance instead of isotropic covariance | LearnedPrior transfer learning · Across datasets | accuracy: 58.7% vs. 56.7%; 83.5% vs. 81.9%; 57.4% vs. 55.5% | Mixed |
| Use initialization-only standard transfer learning instead of the previously reported comparison | CIFAR-10 transfer learning · CIFAR-10, n=1000 | accuracy: 78-82% vs. 64% | Helped |
| Use LearnedPriorLR rather than StdPrior for the target-task loss landscape | CIFAR-10 transfer learning · CIFAR-10, n=1000 | train-test loss alignment | Mixed |
Surprising findings
- Standard transfer learning with an ImageNet initialization performs 10-15 points better than previously reported on CIFAR-10 at n=100 and n=1000.
- The low-rank covariance prior of the prior work is not consistently better than the simpler isotropic learned-mean prior.
Limitations
- Only one source task (ImageNet) and one pre-training snapshot (SimCLR) are studied.
- Only MAP point estimation is studied; full Bayesian SGHMC inference from prior work is not evaluated.
- Hyperparameter search grids were coarser than the original study's for some settings.
Experimental setup
Datasets: CIFAR-10 Oxford Flowers Oxford-IIIT Pets FGVC-Aircraft HAM10000 ImageNet
Benchmarks: CIFAR-10 Oxford Flowers Oxford-IIIT Pets FGVC-Aircraft HAM10000
Metrics: classifier accuracy average AUROC negative log likelihood
Baselines: StdPrior (standard transfer learning) LearnedPriorIso (MAP adaptation) LearnedPriorLR (low-rank SWAG prior)
Builds on: Bayesian transfer learning (Shwartz-Ziv et al. 2022) SWAG SimCLR MAP adaptation
Closest papers by meaning
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.85)
- Wasserstein Transfer Learning (NeurIPS 2025 · similarity 0.84)
- Enhancing Transfer Learning with Flexible Nonparametric Posterior Sampling (ICLR 2024 · similarity 0.84)
- When is Transfer Learning Possible? (ICML 2024 · similarity 0.83)
- Understanding the Transferability of Representations via Task-Relatedness (NeurIPS 2024 · similarity 0.83)
- Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasing (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.