Research map / Architectures and efficiency / Pruning and efficient tuning
DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DropBP, which randomly drops layers during backward propagation only (forward pass intact) with sensitivity-based per-layer drop rates, to speed up and reduce activation memory when fine-tuning LLMs with full FT or PEFT.
Problem
Fine-tuning LLMs with backpropagation remains costly in computation (backward pass ~2x forward FLOPs) and activation memory, which PEFT methods do not address.
Approach
Randomly skip backward propagation through attention/FFN sublayers (keeping forward pass intact) so dropped layers need no gradient FLOPs or stored activations, with per-layer drop rates allocated by a gradient-variance-based sensitivity measure and a greedy solver under a FLOPs budget.
Key results
- DropBP at drop rate 0.875 cuts LoRA training time by 43% and Full-FT by 57% on LLaMA2-7B with comparable accuracy (e.g., LoRA+DropBP 66.4 vs 66.0 commonsense avg).
- Max sequence length for QLoRA fine-tuning LLaMA2-70B on a single A100 rises from 0.6K to 3.7K (6.2x) at drop rate 0.875.
- On A100 with LLaMA3-8B full fine-tuning, throughput increases 79% (16.4 sentences/s) and on Intel Gaudi2 117% (28.4 sentences/s) at drop rate 0.875.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DropBP to LoRA fine-tuning at p=0.875 | LLaMA2-7B on Alpaca · fine-tuning 50K samples | training time: 2.2h -> 1.3h | Helped |
| Add DropBP to full fine-tuning at p=0.875 | LLaMA2-7B on Alpaca · fine-tuning 50K samples | training time: 2.0h -> 0.8h | Helped |
| Add DropBP to QLoRA at p=0.875 | LLaMA2-70B on Alpaca · fine-tuning 50K samples | training time: 29.6h -> 16.5h | Helped |
| Add DropBP at p=0.875 | LLaMA3-8B LoRA fine-tuning on Oasst1 · MT-Bench | average score: 5.12 -> 5.08 | No clear effect |
| Add DropBP at p=0.875 | LLaMA3-8B fine-tuning · LLaMA3-8B Oasst1 | memory: 57G -> 32G | Helped |
| Add DropBP at p=0.875 | LLaMA2-70B QLoRA on NVIDIA A100 · maximum sequence length, micro batch size 1 | maximum sequence length: 0.6K -> 3.7K (6.2×) | Helped |
| Use sensitivity-based rather than uniform layer drop rates (ablation) | LLaMA2-7B LoRA, Alpaca, LR 3e-4 · commonsense reasoning 0-shot | accuracy: 63.1 -> 64.7 | Helped |
| Use sensitivity-based rather than uniform layer drop rates (ablation) | LLaMA2-7B LoRA, Alpaca, LR 1e-4 · commonsense reasoning 0-shot | accuracy: 66.4 -> 66.6 | No clear effect |
| Use sensitivity-based rather than uniform layer drop rates (ablation) | LLaMA2-13B LoRA, Alpaca, LR 1e-4 · commonsense reasoning 0-shot | accuracy: 66.6 -> 67.7 | Helped |
| Randomly drop backward propagation rather than freeze lower layers (ablation) | LLaMA2-7B LoRA, Alpaca, p=0.875 · MMLU 5-shot | accuracy: 42.9 (-1.8) -> 43.7 (-1.0) | Helped |
| Add DropBP at p=0.875 (ablation) | LLaMA2-70B QLoRA, Alpaca · MMLU 5-shot | accuracy: 68.3 -> 68.8 (+0.5) | Helped |
| Use DropBP rather than LayerDrop or Progressive Layer Dropping (ablation) | LLaMA2-7B LoRA · commonsense reasoning 0-shot | accuracy: 58.7 -> 65.9; 61.0 -> 65.9 | Helped |
Surprising findings
- DropBP at drop rate 0.875 slightly improved 5-shot MMLU accuracy for LLaMA2-70B QLoRA (+0.5%) despite skipping most backward layers.
- LayerDrop and PLD, which drop layers in the forward pass, lost over 5% commonsense accuracy at similar relative FLOPs, while DropBP did not.
Limitations
- Convergence per step is slower at high drop rates (0.75, 0.875), so gains depend on per-time rather than per-step convergence.
- Sensitivity-based allocation mainly helps at high learning rates or larger models; uniform drop rate was comparable at low settings.
- Authors note DropBP is designed for fine-tuning and has not been shown to accelerate pretraining.
Experimental setup
Datasets: Alpaca Dolly Oasst1
Benchmarks: MMLU (5-shot) PIQA HellaSwag ARC-C ARC-E OpenBookQA WinoGrande MT-Bench
Metrics: accuracy training time peak memory max sequence length throughput (sentences/s) validation perplexity
Baselines: LoRA QLoRA Full-FT LayerDrop Progressive Layer Dropping layer freezing
Builds on: LoRA QLoRA Stochastic Depth LayerDrop GradNormVar
Closest papers by meaning
- Gradient Weight-normalized Low-rank Projection for Efficient LLM Training (AAAI 2025 · similarity 0.89)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
- Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation (ICLR 2024 · similarity 0.88)
- Enhancing Large Language Model Performance with Gradient-Based Parameter Selection (AAAI 2025 · similarity 0.88)
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.87)
- RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.