Research map / Architectures and efficiency / Pruning and efficient tuning
Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Baohao Liao, Shaomu Tan, Christof Monz
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MEFT_3 instead of full fine-tuning | MEFT_3 · GLUE BERT-base | average: 83.2 -> 84.2 | Improved |
| Use MEFT_1 instead of full fine-tuning | MEFT_1 · GLUE RoBERTa-large | average: 88.9 -> 89.6 | Improved |
| Switch from vanilla to reversible gradient (ablation) | MEFT_3 · GLUE BERT-base | average: 84.2 -> 81.1 | Hurt |
| Switch from vanilla to reversible gradient (FP32) (ablation) | MEFT_1 · GLUE BERT-base | average: 83.8 -> 83.2 | Hurt |
| Use reversible gradient to avoid caching activations (ablation) | MEFT_1 · GLUE BERT-base | activation memory (GB): 14.98 -> 2.33 | Improved |
| Train reversible MEFT_1 in FP32 instead of FP16 (ablation) | MEFT_1 · GLUE RoBERTa-large | average: 87.8 -> 88.9 | Improved |
| Train reversible MEFT_1 in FP32 instead of FP16 (ablation) | MEFT_1 · GLUE BART-large | average: 81.9 -> 83.9 | Improved |
| Use MEFT_1 instead of LST | MEFT_1 · GLUE BART-large | average: 82.1 -> 83.9 | Improved |
| Use MEFT_1 instead of LoRA with OPT-1.3B | MEFT_1 · QA OPT-1.3B | average accuracy: 57.4 -> 57.3 | No clear effect |
| Scale model to OPT-6.7B with MEFT_1 | MEFT_1 · QA | average accuracy: 57.3 -> 61.7 | Improved |
| Make fewer RoBERTa layers reversible (<=16) (ablation) | MEFT · GLUE RoBERTa-large | average score | Improved |
| Freeze shallow layers instead of caching top layers (ablation) | MEFT · GLUE RoBERTa-large | average score | Hurt |
Closest papers by meaning
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.89)
- DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning (ICLR 2024 · similarity 0.89)
- Increasing Model Capacity for Free: A Simple Strategy for Parameter Efficient Fine-tuning (ICLR 2024 · similarity 0.89)
- Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning (ICML 2024 · similarity 0.88)
- Sparse is Enough in Fine-tuning Pre-trained Large Language Models (ICML 2024 · similarity 0.88)
- Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
