AI Paper Atlas

Research map / Label-efficient and robust learning / Self-supervised and contrastive

Meta-learning to Improve Pre-training

Aniruddh Raghu, Jonathan Peter Lorraine, Simon Kornblith, Matthew B.A. McDermott, David Duvenaud

NeurIPS 2021 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
learn per-task pre-training weights via meta-gradients (vs uniform weights)multitask GIN pre-training · PPI, full FT accessmean AUC: 74.7 -> 78.6Improved
learn per-task pre-training weights, 10 of 40 FT tasks held out of meta-PTmultitask GIN pre-training · PPI, partial FT accessmean AUC: 74.8 -> 77.0Improved
co-train on PT and FT data instead of weighted PTmultitask GIN pre-training · PPI, full FT accessmean AUC: 74.7 -> 70.2Hurt
add PCGrad gradient projection to CoTrainCoTrain · PPI, full FT accessmean AUC: 70.2 -> 69.4No clear effect
add PCGrad gradient projection to CoTrain, partial FT accessCoTrain · PPI, partial FT accessmean AUC: 69.3 -> 68.1Hurt
add supervised pre-training on PPI (vs no pre-training)GIN · PPI, full FT accessmean AUC: 66.6 -> 74.7Improved
learn per-example temporal warp strength with a network in SimCLRSimCLR on ECG · PTB-XL, |D_FT|=100mean test AUC: 74.6 -> 76.1Improved
learn per-example temporal warp strength with a network in SimCLRSimCLR on ECG · PTB-XL, |D_FT|=250mean test AUC: 76.5 -> 77.8Improved
learn per-example temporal warp strength with a network in SimCLRSimCLR on ECG · PTB-XL, |D_FT|=500mean test AUC: 79.8 -> 81.7Improved
learn per-example temporal warp strength with a network in SimCLRSimCLR on ECG · PTB-XL, |D_FT|=1000mean test AUC: 82.2 -> 84.0Improved
learn per-example temporal warp strength with a network in SimCLRSimCLR on ECG · PTB-XL, |D_FT|=2500mean test AUC: 85.8 -> 86.7Improved
limit meta-PT FT data to only 50 examples (ablation)Meta-Parameterized SimCLR · PTB-XL, |D_FT|=500test AUC: 79.8 -> 81.3Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy