Research map / Large language models / Instruction and fine-tuning
To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| repeat a small C4 subset for 2^8 epochs rather than train once on a larger dataset | T5 Base · SQuAD | F1: 90.0 -> 88.1 | Hurt |
| use a larger dataset while holding the number of repeats fixed (ablation) | T5 · C4 | masked token prediction accuracy | Helped |
| use Wikipedia rather than C4 as the repeated pre-training dataset (ablation) | T5 · SQuAD | F1 | No clear effect |
| increase trainable parameters while keeping FLOPs comparable (ablation) | T5 with repeated data · C4 | multi-epoch degradation | Hurt |
| increase FLOPs while keeping parameter count comparable (ablation) | T5 with repeated data · C4 | multi-epoch degradation | No clear effect |
| use the UL2 mixture of training objectives instead of the vanilla T5 objective (ablation) | T5 with repeated data · SQuAD | F1 | Hurt |
| add dropout under limited-data training (ablation) | T5 Base · C4 validation | masked token prediction accuracy: 61.7 -> 62.9 | Helped |
| add dropout when training with enough data (ablation) | T5 Base · C4 validation | masked token prediction accuracy: 64.5 -> 63.6 | Hurt |
| add DropPath on top of dropout (ablation) | T5 Base with limited data · C4 validation | masked token prediction accuracy: 62.9 -> 63.0 | No clear effect |
| add label smoothing on top of dropout (ablation) | T5 Base with limited data · C4 validation | masked token prediction accuracy: 62.9 -> 62.6 | No clear effect |
| add weight decay on top of dropout (ablation) | T5 Base with limited data · C4 validation | training stability | Hurt |
| introduce dropout after the first 2^15 steps instead of from the start (ablation) | T5 Large · C4 with repeated data | validation accuracy | Helped |
Closest papers by meaning
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.92)
- When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method (ICLR 2024 · similarity 0.92)
- Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ICLR 2026 · similarity 0.91)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.91)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.91)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.