Research map / Architectures and efficiency / RNNs and normalization
Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, Qun Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Mango instead of LiGO | DeiT-B growth · ImageNet | FLOPs saving ratio (%): 55.7 -> 76.4 | Improved |
| use Mango instead of bert2BERT | DeiT-B growth · ImageNet | FLOPs saving ratio (%): 64.4 -> 76.4 | Improved |
| use Mango instead of bert2BERT | BERT-Base growth · pretraining | FLOPs saving ratio (%): 35.6 -> 39.2 | Improved |
| use Mango instead of bert2BERT | GPT-Base growth · pretraining | acceleration ratio (%): 59.9 | Improved |
| grow from a pretrained smaller model (bert2BERT) vs StackBERT progressive training | BERT-Base · pretraining | FLOPs saving ratio (%): 29.5 -> 35.6 | Improved |
| use Mango growth for DeiT-B instead of training from Scratch | DeiT-B · downstream average | accuracy: 86.82 -> 86.83 | No clear effect |
| use Mango growth for BERT-Base instead of Scratch | BERT-Base · GLUE average | score: 84.45 -> 85.60 | Improved |
| use Mango growth for BERT-Base instead of Scratch | BERT-Base · SQuADv1.1 | F1: 89.21 -> 90.17 | Improved |
| increase Mango operator rank from 1 to 10 (ablation) | Mango · DeiT-T to DeiT-S | operator accuracy and acceleration: 1.19 | No clear effect |
| grow DeiT-T-B vs DeiT-T-A as pretrained model (ablation) | Mango · DeiT-T to DeiT-S | operator accuracy: 72.89 -> 73.81 | Improved |
| expand width, depth, and both with Mango (ablation) | Mango · DeiT-T to DeiT-S | max acceleration ratio (%): 31.0, 46.0, 41.3 | Improved |
| use Mango for Swin-T to Swin-S | Mango · ImageNet | FLOPs saving ratio (%): 58.8 | Improved |
Closest papers by meaning
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.87)
- Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing (NeurIPS 2026 · similarity 0.86)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.85)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.85)
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.85)
- Parameter-Efficient Multi-Task Model Fusion with Partial Linearization (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
