AI Paper Atlas

Research map / Architectures and efficiency / Pruning and efficient tuning

Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

Baohao Liao, Shaomu Tan, Christof Monz

NeurIPS 2023 · Architectures and efficiency › Pruning and efficient tuning · approach: Adapter-based parameter-efficient tuning

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use MEFT_3 instead of full fine-tuningMEFT_3 · GLUE BERT-baseaverage: 83.2 -> 84.2Improved
Use MEFT_1 instead of full fine-tuningMEFT_1 · GLUE RoBERTa-largeaverage: 88.9 -> 89.6Improved
Switch from vanilla to reversible gradient (ablation)MEFT_3 · GLUE BERT-baseaverage: 84.2 -> 81.1Hurt
Switch from vanilla to reversible gradient (FP32) (ablation)MEFT_1 · GLUE BERT-baseaverage: 83.8 -> 83.2Hurt
Use reversible gradient to avoid caching activations (ablation)MEFT_1 · GLUE BERT-baseactivation memory (GB): 14.98 -> 2.33Improved
Train reversible MEFT_1 in FP32 instead of FP16 (ablation)MEFT_1 · GLUE RoBERTa-largeaverage: 87.8 -> 88.9Improved
Train reversible MEFT_1 in FP32 instead of FP16 (ablation)MEFT_1 · GLUE BART-largeaverage: 81.9 -> 83.9Improved
Use MEFT_1 instead of LSTMEFT_1 · GLUE BART-largeaverage: 82.1 -> 83.9Improved
Use MEFT_1 instead of LoRA with OPT-1.3BMEFT_1 · QA OPT-1.3Baverage accuracy: 57.4 -> 57.3No clear effect
Scale model to OPT-6.7B with MEFT_1MEFT_1 · QAaverage accuracy: 57.3 -> 61.7Improved
Make fewer RoBERTa layers reversible (<=16) (ablation)MEFT · GLUE RoBERTa-largeaverage scoreImproved
Freeze shallow layers instead of caching top layers (ablation)MEFT · GLUE RoBERTa-largeaverage scoreHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy