Research map / Label-efficient and robust learning / Self-supervised and contrastive
Meta-learning to Improve Pre-training
Aniruddh Raghu, Jonathan Peter Lorraine, Simon Kornblith, Matthew B.A. McDermott, David Duvenaud
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn per-task pre-training weights via meta-gradients (vs uniform weights) | multitask GIN pre-training · PPI, full FT access | mean AUC: 74.7 -> 78.6 | Improved |
| learn per-task pre-training weights, 10 of 40 FT tasks held out of meta-PT | multitask GIN pre-training · PPI, partial FT access | mean AUC: 74.8 -> 77.0 | Improved |
| co-train on PT and FT data instead of weighted PT | multitask GIN pre-training · PPI, full FT access | mean AUC: 74.7 -> 70.2 | Hurt |
| add PCGrad gradient projection to CoTrain | CoTrain · PPI, full FT access | mean AUC: 70.2 -> 69.4 | No clear effect |
| add PCGrad gradient projection to CoTrain, partial FT access | CoTrain · PPI, partial FT access | mean AUC: 69.3 -> 68.1 | Hurt |
| add supervised pre-training on PPI (vs no pre-training) | GIN · PPI, full FT access | mean AUC: 66.6 -> 74.7 | Improved |
| learn per-example temporal warp strength with a network in SimCLR | SimCLR on ECG · PTB-XL, |D_FT|=100 | mean test AUC: 74.6 -> 76.1 | Improved |
| learn per-example temporal warp strength with a network in SimCLR | SimCLR on ECG · PTB-XL, |D_FT|=250 | mean test AUC: 76.5 -> 77.8 | Improved |
| learn per-example temporal warp strength with a network in SimCLR | SimCLR on ECG · PTB-XL, |D_FT|=500 | mean test AUC: 79.8 -> 81.7 | Improved |
| learn per-example temporal warp strength with a network in SimCLR | SimCLR on ECG · PTB-XL, |D_FT|=1000 | mean test AUC: 82.2 -> 84.0 | Improved |
| learn per-example temporal warp strength with a network in SimCLR | SimCLR on ECG · PTB-XL, |D_FT|=2500 | mean test AUC: 85.8 -> 86.7 | Improved |
| limit meta-PT FT data to only 50 examples (ablation) | Meta-Parameterized SimCLR · PTB-XL, |D_FT|=500 | test AUC: 79.8 -> 81.3 | Improved |
Closest papers by meaning
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.87)
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters (ICML 2016 · similarity 0.85)
- Meta-Learning-Based Surrogate Models for Efficient Hyperparameter Optimization (TPAMI 2025 · similarity 0.85)
- MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters (ICML 2025 · similarity 0.85)
- Memory-Reduced Meta-Learning with Guaranteed Convergence (AAAI 2025 · similarity 0.85)
- CiTrus: Squeezing Extra Performance out of Low-data Bio-signal Transfer Learning (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
