Research map / Architectures and efficiency / Pruning and efficient tuning
Weight Decay Improves Language Model Plasticity
Tessa Han, Sebastian Bordt, Hanlin Zhang, Sham M. Kakade
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise pretraining weight decay above the 0.1 default (to 1.0) (ablation) | Llama-2 and OLMo-2 models at 20 TPP · CoT reasoning fine-tuning (six tasks, average) | downstream accuracy: optimal pretraining weight decay is 1.0 versus default 0.1 | Improved |
| raise pretraining weight decay from 0.1 to 0.3 (ablation) | OLMo-2-1B at 140 TPP · CoT reasoning fine-tuning | downstream accuracy: optimal pretraining weight decay is 0.3 | Improved |
| raise pretraining weight decay from 0.1 to 0.3 (ablation) | OLMo-2-1B at 140 TPP · pretraining validation loss | cross-entropy loss: 2.6088 -> 2.6208 | Hurt |
| raise pretraining weight decay from 0.1 to 1.0 (ablation) | OLMo-2-1B at 140 TPP · pretraining validation loss | cross-entropy loss: 2.6088 -> 2.7064 | Hurt |
| raise pretraining weight decay from 0.1 to 0.5 or 1.0 (ablation) | Llama-2 20 TPP models · pretraining validation loss | cross-entropy loss: optimal weight decay 0.5 (0.5B, 1B), 0.6 (OLMo-2-1B), 1.0 (4B) | Improved |
| raise pretraining weight decay to 10 (ablation) | Llama-2-0.5B at 20 TPP · pretraining validation loss | cross-entropy loss | Hurt |
| raise pretraining weight decay from 0.1 to 0.5 (ablation) | Llama-2-0.5B at 20 TPP · upstream 5-benchmark average (pretrained) | accuracy: 0.40 -> 0.41 | No clear effect |
| raise pretraining weight decay from 0.1 to 1.0 (ablation) | OLMo-2-1B at 20 TPP · upstream 5-benchmark average (pretrained) | accuracy: 0.49 -> 0.50 | No clear effect |
| raise pretraining weight decay from 0.1 to 1.0 (ablation) | OLMo-2-1B at 140 TPP · upstream 5-benchmark average (pretrained) | accuracy: 0.61 -> 0.56 | Hurt |
| raise pretraining weight decay to 1.0 (ablation) | OLMo-2-1B at 20 TPP · attention W_QK rank | pseudo-rank: reduces rank of W_QK by roughly a factor of 2 | Improved |
| raise pretraining weight decay (ablation) | OLMo-2-1B at 20 TPP · train-val gap | loss gap: gap decreases monotonically with weight decay | Improved |
| raise pretraining weight decay (ablation) | all five model groups · linear probing on SST and AG News | probe accuracy: higher at every layer | Improved |
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.90)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.90)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
