Research map / Architectures and efficiency / Pruning and efficient tuning
LEMON: Lossless model expansion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LEMON, a lossless model expansion recipe that initializes larger Transformers from smaller pre-trained ones (with symmetry-breaking width expansion, average expansion for LayerNorm, and lossless depth expansion) plus a faster-decaying learning rate schedule, cutting training cost for ViT and BERT.
Problem
Training large Transformers from scratch wastes the knowledge in already-trained smaller models; existing model expansion methods either break losslessness, suffer symmetry breaking, or cannot handle indivisible width increases or diverse Transformer variants.
Approach
Lossless expansion of width and depth for Transformers that breaks neuron symmetry by using unequal fan-out coefficients, handles indivisible width via average expansion with a rescaled LayerNorm, and pairs this with a learning rate scheduler that uses the same maximum LR but decays faster.
Key results
- LEMON reduces computational cost by 56.7% for ViT(6,512)->ViT(12,768) and ViT(6,384)->ViT(12,768) on ImageNet, matching target accuracy in 130 epochs.
- For BERT, LEMON saves 33.2% compute from BERT(6,512) to BERT(12,768) and 25.5% from BERT(6,384), reaching log MLM loss 1.64.
- On GLUE, BERT(12,768) expanded from BERT(6,384) achieves the best downstream scores among compared settings (e.g., STS-B 0.866 vs 0.744 from scratch).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Expand ViT(6,384) to ViT(12,768) with LEMON instead of training from scratch | Vision Transformer · ImageNet-1k | computational cost savings: 56.7% | Helped |
| Expand ViT(6,512) to ViT(12,768) with LEMON instead of training from scratch | Vision Transformer · ImageNet-1k | computational cost savings: 56.7% | Helped |
| Expand BERT(6,384) to BERT(12,768) with LEMON instead of training from scratch | BERT · English Wiki | computational cost savings: 25.5% | Helped |
| Expand BERT(6,512) to BERT(12,768) with LEMON instead of training from scratch | BERT · English Wiki | computational cost savings: 33.2% | Helped |
| Use a smaller maximum learning rate instead of the default maximum learning rate (ablation) | Expanded ViT(12,768) · ImageNet-1k | validation accuracy | Hurt |
| Use the default maximum learning rate rather than a reduced maximum learning rate (ablation) | Expanded ViT(12,768) · ImageNet-1k | validation accuracy | Helped |
| Use faster learning-rate decay rather than the default longer schedule (ablation) | Expanded ViT(12,768) · ImageNet-1k | epochs to match target-model performance: 130 epochs; 56.67% | Helped |
| Apply the optimized faster-decay training recipe to baseline expansion methods (ablation) | Expanded Vision Transformers · ImageNet-1k | training speed | Helped |
| Use soft knowledge inheritance instead of LEMON | Vision Transformer target model · ImageNet-1k | training speed | No clear effect |
| Use hard knowledge inheritance instead of soft knowledge inheritance | Vision Transformer target model · ImageNet-1k | training speed | Helped |
| Use bert2BERT-FPI or bert2BERT-AKI instead of LEMON | Vision Transformer target model · ImageNet-1k | validation accuracy | Hurt |
| Expand BERT(6,384) rather than BERT(6,512) to BERT(12,768) | LEMON BERT · GLUE | downstream performance: 0.848 -> 0.866; 83.82 -> 85.54; 0.36 -> 0.38; 90.14 -> 90.94; 88.76 -> 89.33; 80.92 -> 81.81; 81.57 -> 81.81; 89.91 -> 9 | Helped |
Surprising findings
- Bert2BERT-FPI and bert2BERT-AKI performed worse than training from scratch in ViT experiments.
- Using a smaller maximum learning rate yields smaller training loss but worse validation accuracy, suggesting generalization rather than optimization is harmed.
Limitations
- Requires tuning the total number of training epochs for the faster-decay schedule.
- Evaluation scale limited by available compute; only ViT and BERT tested.
- Selection of free parameters for initialization not yet systematically developed.
Experimental setup
Datasets: ImageNet-1k English Wikipedia GLUE
Benchmarks: ImageNet-1k English Wiki MLM GLUE
Metrics: validation accuracy MLM validation loss training iterations/epochs to target computational cost savings GLUE accuracy/correlation/MCC
Baselines: Train from scratch bert2BERT-FPI bert2BERT-AKI soft KI hard KI LiGO
Builds on: Net2Net bert2BERT Staged training knowledge inheritance DeiT
Closest papers by meaning
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.87)
- Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective (AAAI 2025 · similarity 0.86)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.85)
- PELA: Learning Parameter-Efficient Models with Low-Rank Approximation (CVPR 2024 · similarity 0.85)
- Deep Fusion: Efficient Network Training via Pre-trained Initializations (ICML 2024 · similarity 0.85)
- LAuReL: Learned Augmented Residual Layer (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.