Research map / Label-efficient and robust learning / Recognition and long-tail
Insights into Pre-training via Simpler Synthetic Tasks
Yuhuai Wu, Felix Li, Percy Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on LIME synthetic task instead of random init | T5-Small · CNNDM-10K | ROUGE1: 18.9 -> 33.2 | Improved |
| Pre-train on LIME instead of random init | T5-Small · SQuAD | EM: 17.2 -> 50.4 | Improved |
| Pre-train on the Set task instead of random init | T5-Small · Six-task average | fraction of natural pre-training benefit: 65% | Improved |
| Pre-train on Identity (copy) task instead of random init | T5-Small · CNNDM-10K | ROUGE1: 18.9 -> 30.2 | Improved |
| Pre-train on Identity instead of Set | T5-Small · SQuAD | EM: 48.3 -> 26.1 | Hurt |
| Use Wikipedia 10K-step natural pre-training instead of LIME 30K-step | T5-Small · Six-task average | average score: 55.4 -> 58.9 | Improved |
| Natural language Wiki 10K instead of synthetic pre-training | T5-Small · Code Translation | EM: 59.0 -> 60.2 | Improved |
| Initialize with per-parameter mean and SD of Set pre-trained weights instead of the weights (ablation) | T5-Small · MTOP | EM: 38.6 -> 66.6 | Improved |
| Initialize with per-parameter scale statistic only instead of mean and SD (ablation) | T5-Small · CNNDM-10K | ROUGE1: 29.8 -> 29.5 | No clear effect |
| Use whole-model scale statistics (two numbers) instead of per-parameter (ablation) | T5-Small · CNNDM-10K | ROUGE1: 29.3 -> 15.5 | Hurt |
| Initialize only pre-attention layer norms from pre-trained mean (ablation) | T5-Small · MTOP | EM: 38.6 -> 59.9 | Improved |
| Initialize pre-attention layer norm to 0.2 instead of 1.0 (ablation) | T5-Small · CNNDM-10K | ROUGE1: 18.9 -> 28.2 | Improved |
Closest papers by meaning
- On the Provable Advantage of Unsupervised Pretraining (ICLR 2024 · similarity 0.86)
- Train Once, Answer All: Many Pretraining Experiments for the Cost of One (ICLR 2026 · similarity 0.86)
- Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining (NeurIPS 2026 · similarity 0.86)
- How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? (ICLR 2024 · similarity 0.85)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.85)
- Pre-training with Synthetic Data Helps Offline Reinforcement Learning (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
