Research map / Architectures and efficiency / Pruning and efficient tuning
Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Approx-BP and Memory-Sharing BP to cut fine-tuning activation memory by replacing GELU/SiLU derivatives with 2-bit ReLU-based step functions (ReGELU2/ReSiLU2) and sharing activation memory in LayerNorm/RMSNorm (MS-LN/MS-RMSNorm), reducing peak memory ~30% without throughput loss.
Problem
Fine-tuning large pretrained transformers is limited by activation memory, and non-linear modules (activation functions and layer normalization) account for a large share of it, while prior reduction methods (checkpointing, quantization) cost training throughput.
Approach
Decouple forward and backward passes: keep the pretrained forward activation (GELU/SiLU) but use a 4-segment step-function derivative fit by combinations of ReLUs (needing only 2 bits per element), and merge LayerNorm/RMSNorm affine parameters into the following linear layer so that the norm input can be shared with the next layer's activation memory.
Key results
- ReGELU2 + MS-LN with LoRA (r=4) on ViT-base reduces peak memory by about 29-30% (2717 vs 3827 MiB on Q,V adaptation) with throughput unchanged and accuracy maintained.
- ReSiLU2 + MS-RMSNorm with QLoRA reduces peak memory by 29% on LLaMA-7B (14.6 vs 20.6 GiB) and LLaMA-13B (22.3 vs 31.4 GiB), with about 10% higher throughput and MMLU accuracy drop of 0.40% and 0.14%.
- On RoBERTa-base with LoRA on GLUE, ReGELU2 + MS-LN reduces memory by about 21% (5173 vs 6517 MiB) with mean accuracy 81.73 vs 80.41.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace GELU and LN with ReGELU2 and MS-LN | LoRA r=4 on ViT-base · CIFAR10/100 and FGVC, adapt Q,V | Mem.(MiB): 3827 -> 2717; -29% | Helped |
| Replace GELU and LN with ReGELU2 and MS-LN | LoRA r=4 on ViT-base · CIFAR10/100 and FGVC, adapt all linear | Mem.(MiB): 5128 -> 3601; -30% | Helped |
| Replace GELU and LN with ReGELU2 and MS-LN | Full Tuning ViT-base · CIFAR10/100 and FGVC | Mem.(GiB): 5.6 -> 4.1; -27% | Helped |
| Replace GELU and LN with ReGELU2 and MS-LN | Full Tuning ViT-large · CIFAR10/100 and FGVC | Mem.(GiB): 15.7 -> 11.5; -27% | Helped |
| Replace SiLU and RMSNorm with ReSiLU2 and MS-RMSNorm | QLoRA r=64 on LLaMA-7B · 5-shot MMLU after Alpaca fine-tuning | Mem.(GiB): 20.6 -> 14.6; -29% | Helped |
| Replace SiLU and RMSNorm with ReSiLU2 and MS-RMSNorm | QLoRA r=64 on LLaMA-13B · 5-shot MMLU after Alpaca fine-tuning | Mem.(GiB): 31.4 -> 22.3; -29% | Helped |
| Replace SiLU and RMSNorm with ReSiLU2 and MS-RMSNorm | QLoRA r=64 on LLaMA-7B · 5-shot MMLU after Alpaca fine-tuning | Accuracy(%): 40.75 -> 40.35 | Hurt |
| Replace SiLU and RMSNorm with ReSiLU2 and MS-RMSNorm | QLoRA r=64 on LLaMA-13B · 5-shot MMLU after Alpaca fine-tuning | Accuracy(%): 46.68 -> 46.54 | Hurt |
| Replace GELU and LN with ReGELU2 and MS-LN | LoRA r=64 Q,K on RoBERTa-base · GLUE mean | Accuracy(%): 80.41 -> 81.73 | Helped |
| Replace GELU and LN with ReGELU2 and MS-LN | LoRA r=64 Q,K on RoBERTa-base · GLUE mean | Mem.(MiB): 6517 -> 5173; -21% | Helped |
| Use ReGELU2 instead of GELU while retaining LN (ablation) | LoRA r=4 on ViT-base · CIFAR10/100 and FGVC, adapt Q,V | Mem.(MiB): 3827 -> 3087; -19% | Helped |
| Use MS-LN instead of LN while retaining GELU (ablation) | LoRA r=4 on ViT-base · CIFAR10/100 and FGVC, adapt all linear | Mem.(MiB): 5128 -> 4316; -16% | Helped |
Surprising findings
- Replacing the forward pass of SiLU with the ReLU-combination approximation severely degrades no-tuning MMLU accuracy (35.62% to 23.44% for LLaMA-7B), so only the backward pass is changed.
- Mesa 8-bit quantization degrades training throughput (e.g., 288 to 245 images/s), while the proposed method keeps throughput.
Limitations
- MS-LN cannot share memory when LoRA-FA is used or when the following FFN linear layers are frozen, so gains are smaller in those settings.
- The Approx-BP convergence guarantee relies on assumptions, and ReLU-based derivatives violate smoothness only on a zero-measure set; the paper also notes the approach is tied to specific activations (GELU/SiLU).
Experimental setup
Datasets: ImageNet-22k CIFAR10 CIFAR100 FGVC Alpaca GLUE MMLU
Benchmarks: CIFAR10 CIFAR100 FGVC MMLU GLUE
Metrics: top-1 accuracy peak memory (MiB/GiB) training throughput (images/s, samples/s)
Baselines: Mesa gradient checkpointing LoRA-FA QLoRA full fine-tuning LoRA
Builds on: LoRA LoRA-FA QLoRA Mesa FlashAttention
Closest papers by meaning
- Backprop with Approximate Activations for Memory-efficient Network Training (NeurIPS 2019 · similarity 0.91)
- Sheared Backpropagation for Fine-tuning Foundation Models (CVPR 2024 · similarity 0.89)
- Accelerated training through iterative gradient propagation along the residual path (ICLR 2025 · similarity 0.87)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.87)
- LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning (NeurIPS 2024 · similarity 0.86)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.