Research map / Architectures and efficiency / RNNs and normalization
Memory Savings at What Cost? A Study of Alternatives to Backpropagation
Kunjal Panchal, Sunav Choudhary, Yuriy Brun, Hui Guan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BP with checkpointing instead of ZO-Vanilla | fine-tuning · BoolQ (Llama 3.1 8B) | test accuracy (%): 57.1 -> 88.3 | Improved |
| use BP with checkpointing instead of FMAD-Vanilla | fine-tuning · AGNews (Llama 3.1 8B) | test accuracy (%): 80.5 -> 94.2 | Improved |
| use FMAD instead of ZO (analytic jvp vs finite difference) | gradient estimation · AGNews (Llama 3.1 8B) | test accuracy (%): 73.6 -> 80.5 | Improved |
| add gradient accumulation (ablation) | FMAD-Vanilla · BoolQ (Llama 3.1 8B) | test accuracy (%): 60.7 -> 70.3 | Improved |
| add gradient accumulation (ablation) | FMAD-Vanilla · GSM8K (Llama 3.1 8B) | test accuracy (%): 37.7 -> 30.8 | Hurt |
| average over multiple perturbations per iteration (n=10, 50) (ablation) | ZO and FMAD · AGNews (Llama 3.1 8B) | test accuracy: +13.2-14.0% (ZO), +5.7-7.7% (FMAD) | Improved |
| increase perturbations to 10 per iteration in ZO (ablation) | ZO-Vanilla · AGNews (Llama 3.1 8B) | compute cost: 9.7x compute | Hurt |
| accumulation window of 200 (ablation) | ZO and FMAD · AGNews (Llama 3.1 8B) | convergence time: 14.8x (FMAD), 10.9x (ZO) slower | Hurt |
| add adaptive perturbation sampling (ablation) | FMAD-Vanilla · BoolQ and MultiRC (Llama 3.1 8B) | test accuracy (%): -4.3% BoolQ, -3.2% MultiRC | Hurt |
| add sparse parameter updates (top-1%) (ablation) | FMAD and ZO · Llama 3.1 8B tasks | test accuracy: -1.2 to -10.1% (FMAD), -1.9 to -9.0% (ZO) | Hurt |
| add SVRG variance reduction (ablation) | FMAD and ZO · classification tasks (Llama 3.1 8B) | test accuracy: +4.0 to +11.1% classification, -4.2% GSM8K | Mixed |
| use BP-Checkpointing instead of ZO-Vanilla in total compute | training cost · AGNews (Llama 3.1 8B) | TFLOPs until convergence: 251.2 -> 65.2 (x10^4) | Improved |
Closest papers by meaning
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.91)
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.90)
- Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning (NeurIPS 2025 · similarity 0.90)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.89)
- StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMs (NeurIPS 2025 · similarity 0.89)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
