Research map / Large language models / Language, safety and interpretability
Reformulation for Pretraining Data Augmentation
Xintong Hao, Rui-Jie Zhu, Ge Zhang, Ke Shen, Chenggang Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add MGA reformulated data to the pretraining mix | 134M model · LightEval average | average accuracy: 31.51 -> 31.77 | Improved |
| add MGA reformulated data to the pretraining mix | 377M model · LightEval average | average accuracy: 34.57 -> 35.52 | Improved |
| add MGA reformulated data to the pretraining mix | 1.7B model · LightEval average | average accuracy: 41.15 -> 43.4 | Improved |
| add MGA reformulated data to the pretraining mix | 1.7B model · TriviaQA | accuracy: 4.95 -> 20.42 | Improved |
| expand 50B data with 200B MGA tokens instead of 10x repetition | 13B model, EntireSet · benchmark average at 200-500B tokens | average accuracy: +2.65/+3.14/+3.43/+3.46 | Improved |
| collect more unique data (Full-Fineweb-Edu 195B) instead of repeating 50B | 13B model, EntireSet · benchmark average at 200-500B tokens | average accuracy: +0.2/+0.15/-0.16/+0.11 | No clear effect |
| add MGA expansion instead of upsampling high-quality portion | 377M to 13B, Subset · benchmark average | average accuracy: +1.46/+2.67/+3.59/+3.73 vs +0.89/+1.53/+1.23/+1.41 | Improved |
| use MGA instead of Cosmopedia as synthetic source | 377M model · synthetic data comparison | average accuracy: 35.57 -> 37.28 | Improved |
| use Nemotron diverse QA pairs instead of MGA | 377M model · synthetic data comparison | average accuracy: 37.34 -> 40.72 | Hurt |
| replace all real fineweb-edu data with MGACorpus (MGA-Only) (ablation) | 1.7B model · LightEval average | average accuracy: 43.40 -> 42.45 | Hurt |
| use strict information-preserving prompt (SLM-Strict) instead of base (ablation) | MGA reformulator · high-repetition pretraining | validation loss scaling: 78.37% vs 71.06% score>=4 | Mixed |
| use relaxed prompt (SLM-Relaxed) instead of base (ablation) | MGA reformulator · high-repetition pretraining | benchmark performance: 13.63% vs 71.06% score>=4 | Hurt |
Closest papers by meaning
- Automating Dataset Updates Towards Reliable and Timely Evaluation of Large Language Models (NeurIPS 2024 · similarity 0.87)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.87)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.87)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.86)
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.86)
- Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
