Research map / Architectures and efficiency / Pruning and efficient tuning
Sheared Backpropagation for Fine-tuning Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PreBackRazor, a joint activation-and-gradient sparsified backpropagation method using a bandit-updated column-wise freezing probability, to cut memory and compute of fine-tuning ViT and BERT on consumer GPUs.
Problem
Activation pruning for memory-efficient fine-tuning (e.g., BackRazor) reduces activation memory but its element-wise top-k pruning adds substantial computation overhead, making fine-tuning slower on low-memory devices.
Approach
Replace element-wise self-sparsified top-k activation pruning with column-wise joint sparsification: a bandit (JointSpar-style) updates per-coordinate freezing probabilities from gradient norms, so non-essential activations are never pruned or stored, and the FlashAttention-2 kernel is combined with the method.
Key results
- On ViT-B/16 fine-tuning CIFAR-100, PreBR++JS@50% reduces peak memory from 7540MB (BackRazor) to 6880MB with 1.6x speedup and 92.1% accuracy vs 93.0% full fine-tuning.
- Integrating with FlashAttention yields 1.7x training speedup over vanilla attention (3min40s vs 6min19s epoch time for BackRazor) at 92.2% CIFAR-100 accuracy.
- On BERT RTE with batch size 8 at 80% pruning, PreBackRazor reaches 72.92% accuracy versus 70.04% for full fine-tuning, with up to 9.7% speedup over BackRazor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PREBR++JS@50% instead of BACKRAZOR | ViT-B/16 fine-tuning · CIFAR-100 | accuracy: 92.1 -> 92.1 | No clear effect |
| Use PREBR++JS@50% instead of BACKRAZOR | ViT-B/16 fine-tuning · CIFAR-10 | accuracy: 98.6 -> 98.6 | No clear effect |
| Use PREBR++JS@50% instead of BACKRAZOR | ViT-B/16 fine-tuning · CUB-200 | accuracy: 86.6 -> 85.6 | Hurt |
| Use PREBR++JS@50% instead of BACKRAZOR | ViT-B/16 fine-tuning · CIFAR-100 | epoch time: 6min19s -> 4min02s | Helped |
| Replace BACKRAZOR attention with FLASHATTENTION (ablation) | PREBR++JS for ViT-B/16 · CIFAR-100 | speedup: 1.0× -> 1.7× | Helped |
| Use FLASHATTENTION instead of BACKRAZOR attention (ablation) | PREBR++JS for ViT-B/16 · CIFAR-100 | accuracy: 91.4 -> 92.2 | Helped |
| Use sparse FLASHATTN instead of FLASHATTENTION (ablation) | PREBR++JS for ViT-B/16 · CIFAR-100 | accuracy: 92.2 -> 92.1 | Hurt |
| Use PREBR instead of BACKRAZOR at 80% pruning | bert-base-uncased fine-tuning · RTE, GLUE; batch size 8 | accuracy: 69.31 -> 72.92 | Helped |
| Use PREBR instead of BACKRAZOR at 90% pruning | bert-base-uncased fine-tuning · RTE, GLUE; batch size 8 | accuracy: 66.43 -> 69.68 | Helped |
| Use PREBR instead of BACKRAZOR at 80% pruning | bert-base-uncased fine-tuning · RTE, GLUE; batch size 32 | accuracy: 65.70 -> 66.07 | Helped |
| Use PREBR instead of BACKRAZOR at 90% pruning | bert-base-uncased fine-tuning · RTE, GLUE; batch size 32 | accuracy: 66.43 -> 65.70 | Hurt |
| Use PREBR instead of BACKRAZOR at 80% pruning | bert-base-uncased fine-tuning · RTE, GLUE; batch size 8 | epoch completion time: 17min39s -> 16min36s | Helped |
Surprising findings
- Element-wise self-sparsified top-k pruning in BackRazor can increase training time up to 5x at batch size 8 on BERT, with only about 12% memory reduction.
- Column-wise sparsity gives up to 1.7x speedup on a linear layer over element-wise sparsity at the same ratio, despite the pattern being less fine-grained.
Limitations
- Theoretical convergence analysis is restricted to linear networks.
- Experiments only on ViT-B/16 and BERT-base; applicability to larger foundation models not tested.
- Mixed results on accuracy; some configurations drop accuracy (e.g., PREBR at 90% on BERT RTE batch 32).
Experimental setup
Datasets: ImageNet-22K CIFAR-10 CIFAR-100 CUB-200 RTE (GLUE)
Benchmarks: CIFAR-10 CIFAR-100 CUB-200 GLUE RTE
Metrics: accuracy peak GPU memory epoch/wall-clock time speedup
Baselines: BackRazor Full fine-tuning JointSpar
Builds on: BackRazor JointSpar FlashAttention-2
Closest papers by meaning
- Reducing Fine-Tuning Memory Overhead by Approximate and Memory-Sharing Backpropagation (ICML 2024 · similarity 0.89)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.89)
- RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning (NeurIPS 2026 · similarity 0.87)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.86)
- PROFIT: A Specialized Optimizer for Deep Fine Tuning (NeurIPS 2025 · similarity 0.86)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.