Research map / Label-efficient and robust learning / Self-supervised and contrastive
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, Andrew Gordon Wilson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an informative learned prior instead of a non-learned prior | Bayesian transfer with SimCLR ResNet-50 · CIFAR-10, 10 samples | test error: 86.1± 3.9 -> 73.8± 3.2 | Improved |
| Use Bayesian inference with the learned prior rather than SGD | SimCLR ResNet-50 transfer · CIFAR-100, 10 samples | test error: 90.9± 2.8 -> 81.9± 3.7 | Improved |
| Use a learned SSL prior rather than a learned supervised prior | Bayesian segmentation · PASCAL VOC 2012 val, ResNet-50 | Mean-IoU: 73.72 -> 74.15 | Improved |
| Use a learned SSL prior rather than a learned supervised prior | Bayesian segmentation · Cityscapes val, ResNet-50 | Mean-IoU: 76.09 -> 76.52 | Improved |
| Replace the learned covariance with an isotropic covariance while retaining the learned mean (ablation) | Bayesian transfer with learned prior · CIFAR-100 | performance | Hurt |
| Remove the low-rank covariance component (ablation) | SWAG learned prior · CIFAR-10 and CIFAR-100 | performance | Hurt |
| Use rank-5 rather than rank-0 low-rank covariance (ablation) | SWAG learned prior · CIFAR-10 | performance | Improved |
| Increase the low-rank covariance rank through 7 (ablation) | SWAG learned prior · CIFAR-100 | performance | Improved |
| Rescale the learned prior covariance to be more diffuse for the downstream task (ablation) | Bayesian transfer with learned prior · CIFAR-10 | accuracy | Improved |
| Use Bayesian model averaging rather than MAP estimation (ablation) | Transfer with learned prior · Downstream classification tasks | performance | Improved |
| Use a learned prior rather than standard pre-trained initialization | SGD transfer learning · Semantic segmentation | Mean-IoU: 73.17 -> 73.48 | Improved |
Closest papers by meaning
- Deep Reference Priors: What is the best way to pretrain a model? (ICML 2022 · similarity 0.90)
- Transfer Learning with Informative Priors: Simple Baselines Better than Previously Reported (NeurIPS 2024 · similarity 0.87)
- Easy Bayesian Transfer Learning with Informative Priors (NeurIPS 2023 · similarity 0.87)
- Transferring Knowledge From Large Foundation Models to Small Downstream Models (ICML 2024 · similarity 0.85)
- Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining (NeurIPS 2026 · similarity 0.84)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
