AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Reusing Pretrained Models by Multi-linear Operators for Efficient Training

Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, Qun Liu

NeurIPS 2023 · Architectures and efficiency › RNNs and normalization · approach: Dataset distillation and data-efficient training · 4 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use Mango instead of LiGODeiT-B growth · ImageNetFLOPs saving ratio (%): 55.7 -> 76.4Improved
use Mango instead of bert2BERTDeiT-B growth · ImageNetFLOPs saving ratio (%): 64.4 -> 76.4Improved
use Mango instead of bert2BERTBERT-Base growth · pretrainingFLOPs saving ratio (%): 35.6 -> 39.2Improved
use Mango instead of bert2BERTGPT-Base growth · pretrainingacceleration ratio (%): 59.9Improved
grow from a pretrained smaller model (bert2BERT) vs StackBERT progressive trainingBERT-Base · pretrainingFLOPs saving ratio (%): 29.5 -> 35.6Improved
use Mango growth for DeiT-B instead of training from ScratchDeiT-B · downstream averageaccuracy: 86.82 -> 86.83No clear effect
use Mango growth for BERT-Base instead of ScratchBERT-Base · GLUE averagescore: 84.45 -> 85.60Improved
use Mango growth for BERT-Base instead of ScratchBERT-Base · SQuADv1.1F1: 89.21 -> 90.17Improved
increase Mango operator rank from 1 to 10 (ablation)Mango · DeiT-T to DeiT-Soperator accuracy and acceleration: 1.19No clear effect
grow DeiT-T-B vs DeiT-T-A as pretrained model (ablation)Mango · DeiT-T to DeiT-Soperator accuracy: 72.89 -> 73.81Improved
expand width, depth, and both with Mango (ablation)Mango · DeiT-T to DeiT-Smax acceleration ratio (%): 31.0, 46.0, 41.3Improved
use Mango for Swin-T to Swin-SMango · ImageNetFLOPs saving ratio (%): 58.8Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy