Research map / Architectures and efficiency / Pruning and efficient tuning
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train larger Transformers and stop before convergence | Transformer pretraining and WMT14 English→French · Full pretraining corpus and WMT14 English→French | wall-clock time to convergence | Helped |
| Increase model size on the full pretraining dataset | RoBERTa · Full pretraining corpus | validation perplexity | Helped |
| Increase model size on the 5% data subset (ablation) | RoBERTa · 5% pretraining data | wall-clock time to convergence | No clear effect |
| Increase model size on the 1% data subset (ablation) | RoBERTa · 1% pretraining data | validation perplexity | Hurt |
| Increase batch size beyond 2048, with its associated learning-rate tuning (ablation) | 12-layer, 768H RoBERTa · Pretraining | wall-clock time to convergence | No clear effect |
| Train a larger model rather than increasing batch size in the 2048–16384 range (ablation) | RoBERTa pretraining · Pretraining | wall-clock training efficiency | Helped |
| Increase model size while holding pretraining perplexity roughly equal (ablation) | RoBERTa finetuned on MNLI and SST-2 · MNLI and SST-2 | accuracy: 84.3 -> 85.4; 93.0 -> 92.6 | Mixed |
| Quantize large models more heavily rather than lightly compressing small models | Finetuned RoBERTa · MNLI | accuracy at comparable memory budgets | Helped |
| Prune large models more heavily rather than pruning small models | Finetuned RoBERTa · MNLI | accuracy at comparable nonzero-parameter budgets | Helped |
| Combine pruning with quantization (ablation) | Finetuned RoBERTa · MNLI | accuracy-efficiency trade-off | Helped |
| Use 4-bit quantization across different pretraining checkpoints (ablation) | RoBERTa finetuned on MNLI · MNLI | accuracy drop from quantization | No clear effect |
| Share all Transformer layers or only attention layers (ablation) | RoBERTa · Pretraining | convergence and final performance | Hurt |
Closest papers by meaning
- Language Modeling Is Compression (ICLR 2024 · similarity 0.90)
- Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers (NeurIPS 2025 · similarity 0.90)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.90)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.89)
- Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.89)
- Compressed Context Memory for Online Language Model Interaction (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.