AI Paper Atlas

Research map / Label-efficient and robust learning / Recognition and long-tail

Insights into Pre-training via Simpler Synthetic Tasks

Yuhuai Wu, Felix Li, Percy Liang

NeurIPS 2022 · Label-efficient and robust learning › Recognition and long-tail · approach: Contrastive representation theory

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Pre-train on LIME synthetic task instead of random initT5-Small · CNNDM-10KROUGE1: 18.9 -> 33.2Improved
Pre-train on LIME instead of random initT5-Small · SQuADEM: 17.2 -> 50.4Improved
Pre-train on the Set task instead of random initT5-Small · Six-task averagefraction of natural pre-training benefit: 65%Improved
Pre-train on Identity (copy) task instead of random initT5-Small · CNNDM-10KROUGE1: 18.9 -> 30.2Improved
Pre-train on Identity instead of SetT5-Small · SQuADEM: 48.3 -> 26.1Hurt
Use Wikipedia 10K-step natural pre-training instead of LIME 30K-stepT5-Small · Six-task averageaverage score: 55.4 -> 58.9Improved
Natural language Wiki 10K instead of synthetic pre-trainingT5-Small · Code TranslationEM: 59.0 -> 60.2Improved
Initialize with per-parameter mean and SD of Set pre-trained weights instead of the weights (ablation)T5-Small · MTOPEM: 38.6 -> 66.6Improved
Initialize with per-parameter scale statistic only instead of mean and SD (ablation)T5-Small · CNNDM-10KROUGE1: 29.8 -> 29.5No clear effect
Use whole-model scale statistics (two numbers) instead of per-parameter (ablation)T5-Small · CNNDM-10KROUGE1: 29.3 -> 15.5Hurt
Initialize only pre-attention layer norms from pre-trained mean (ablation)T5-Small · MTOPEM: 38.6 -> 59.9Improved
Initialize pre-attention layer norm to 0.2 instead of 1.0 (ablation)T5-Small · CNNDM-10KROUGE1: 18.9 -> 28.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy