Research map / Large language models / Instruction and fine-tuning
Towards Green AI in Fine-tuning Large Language Models via Adaptive Backpropagation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GreenTrainer, which adaptively selects which LLM tensors to fine-tune each epoch by combining a first-order Taylor tensor importance metric with a backpropagation-FLOPs-aware dynamic programming selector, cutting training FLOPs by up to 64% with little accuracy loss.
Problem
Fine-tuning LLMs consumes large compute and carbon, but existing parameter-efficient methods (LoRA, prefix tuning, fixed layer selection) reduce only weight-update cost and still backpropagate activation gradients through the whole model, limiting FLOPs savings.
Approach
Select trainable tensors under an explicit FLOPs budget by modeling cross-tensor backpropagation dependencies in FLOPs, scoring each tensor's importance via first-order Taylor expansion of the loss (Δw · ∂L/∂w), and solving the resulting constrained selection with dynamic programming.
Key results
- GreenTrainer saves up to 64% training FLOPs versus full fine-tuning on OPT-2.7B (GT-0.36 on SciTLDR, 14.9 vs 41.8 PFLOPs) with ROUGE that is substantially lower at that budget.
- At a 50% FLOPs budget, GT-0.5 on OPT-2.7B reaches 30.5/13.1/25.2 ROUGE on SciTLDR versus 32.9/14.9/27.1 for Full FT, with 50% fewer FLOPs.
- GT-0.7 on OPT-2.7B reaches 33.1/15.2/27.6 on SciTLDR, slightly above Full FT, with 30% fewer FLOPs, and outperforms LoRA at similar FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive backpropagation with ρ=0.5 instead of full fine-tuning | OPT-2.7B · SciTLDR | ROUGE-1: 32.9 -> 30.5 | Hurt |
| use adaptive backpropagation with ρ=0.7 instead of full fine-tuning | OPT-2.7B · SciTLDR | ROUGE-1: 32.9 -> 33.1 | Helped |
| use adaptive backpropagation with ρ=0.7 instead of LoRA | OPT-2.7B · DialogSum | ROUGE-1: 23.8 -> 26.8 | Helped |
| use update-gradient tensor importance instead of update magnitude (ablation) | GreenTrainer · SciTLDR | ROUGE-1: 32.7 -> 33.1 | Helped |
| use update-gradient tensor importance instead of gradient-only importance (ablation) | GreenTrainer · DialogSum | ROUGE-1: 25.0 -> 26.8 | Helped |
| use ρ=0.36 instead of ρ=0.4 (ablation) | GreenTrainer on OPT-2.7B · SciTLDR | ROUGE-1: 28.6 -> 4.1 | Hurt |
| increase tensor-importance evaluation frequency from once per epoch to every 600 iterations (ablation) | GreenTrainer on OPT-2.7B · WebQuestion | Accuracy: 28.4 -> 28.5 | No clear effect |
| increase tensor-importance evaluation frequency from once per epoch to every 200 iterations (ablation) | GreenTrainer on OPT-2.7B · WebQuestion | Accuracy: 28.4 -> 27.5 | Hurt |
| use dynamic tensor selection once per epoch instead of fixed selection in the first epoch (ablation) | GreenTrainer on OPT-2.7B · WebQuestion | Accuracy: 27.4 -> 28.4 | Helped |
| increase tensor selection frequency from once per epoch to five times per epoch (ablation) | GreenTrainer on OPT-2.7B · WebQuestion | Accuracy: 28.4 -> 27.5 | Hurt |
| fine-tune only the last two layers instead of all parameters (ablation) | OPT-2.7B · DialogSum | ROUGE-1: 23.6 -> 20.8 | Hurt |
| fine-tune decoder prefixes instead of all parameters (ablation) | OPT-2.7B · DialogSum | ROUGE-1: 23.6 -> 13.4 | Hurt |
Surprising findings
- GT-0.7 on OPT-2.7B slightly exceeds full fine-tuning ROUGE on SciTLDR, which the authors attribute to reduced overfitting.
- Fixed tensor selection (FT-Top2, Prefix-T) can lose up to about 10 ROUGE points on some settings, showing that naive fixed substructure selection is harmful.
Limitations
- Accuracy drops of 3-4% at the most aggressive FLOPs budgets (rho <= 0.4), so the trade-off is not free.
- Full FT and GT-0.7 on OPT-6.7B with DialogSum ran out of memory on the GPUs used, limiting evaluation at the largest scale.
- Tensor importance is estimated on the training data and evaluated at epoch granularity with a first-order approximation.
Experimental setup
Datasets: SciTLDR DialogSum WebQuestions PIQA
Benchmarks: SciTLDR DialogSum WebQuestions PIQA
Metrics: ROUGE-1/2/L accuracy (exact match) PFLOPs wall-clock time
Baselines: Full FT FT-Top2 Prefix Tuning LoRA
Builds on: LoRA Prefix Tuning SNIP Fisher pruning
Closest papers by meaning
- Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptation (NeurIPS 2026 · similarity 0.88)
- Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning (NeurIPS 2025 · similarity 0.88)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.88)
- DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation (NeurIPS 2024 · similarity 0.88)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.88)
- Linearization Explains Fine-Tuning in Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.