Research map / Large language models / Language, safety and interpretability
Scaling with Collapse: Efficient and Predictable Training of LLM Families
Shane Bergsma, Bin Claire Zhang, Nolan Simran Dey, Shaheer Muhammad, Gurpreet Gosal, Joel Hestness
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| match LR schedule, TPP and AdamW timescale tau across model sizes | muP LLM families · SlimPajama, 111M-3.3B | normalized training loss curve collapse | Improved |
| mis-scale tau across sizes (Llama-2 style families) (ablation) | LLM families · Llama-2 curves | normalized loss curve collapse | Hurt |
| increase tau (AdamW timescale) (ablation) | 610M, 80 TPP · SlimPajama | TLC shape | Mixed |
| increase TPP at fixed tau (ablation) | 111M and 610M · SlimPajama | TLC shape | Improved |
| use CompleteP instead of muP (ablation) | Celerity · Celerity pretraining | efficiency and reliability | Improved |
| emphasize educational, math and coding data instead of general SlimPajama (ablation) | Celerity · Celerity pretraining | downstream quality | Improved |
| monitor collapse residuals against a small-model reference | Celerity 1.8B, 234 TPP run · Training diagnostics | time to detect divergence: divergence pinpointed near 60% vs raw loss rise only after 90% | Improved |
| fix tau (adjust weight decay) rather than fixed weight decay in batch size sweeps (ablation) | Batch size sweeps · Hyperparameter tuning | curve ordering for early stopping | Improved |
| fit b and q as power laws in tau and TPP instead of fixed values (ablation) | Normalized TLC surrogate model · Surrogate fit, 111M data | MAE: MAE reduced by two-thirds | Improved |
| fit b and q jointly on both tau and TPP (ablation) | Normalized TLC surrogate model · Surrogate fit | MAE | No clear effect |
| use predicted-best (collapse-aligned) early stopping vs current-best | Weight decay sweeps · 1.7B/20TPP and 3.3B/30TPP | final loss gap of chosen setting: negligible gap after 30% and 10% of training | Improved |
| use current-best selection at stop point instead of predicted-best (ablation) | Weight decay sweeps · 1.7B/20TPP vs 3.3B/30TPP | final loss gap | Mixed |
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.91)
- Scaling Optimal LR Across Token Horizons (ICLR 2025 · similarity 0.90)
- On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.88)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.88)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.87)
- Strong Model Collapse (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
