Research map / Architectures and efficiency / Pruning and efficient tuning
Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSP-Offload, a CPU-GPU offloading framework that fine-tunes LLMs by updating in a subspace defined by learned (d,r)-sparse projectors plus a layer-wise overlapping communication schedule.
Problem
Fine-tuning LLMs exceeds single commodity GPU memory; existing CPU offloading is bounded by limited PCIe bandwidth and slow CPU compute, while PEFT methods (LoRA, GaLore) have memory and compute that scale linearly with subspace rank and can converge slowly.
Approach
Constrain updates to a periodically refreshed subspace defined by sparse projectors whose GPU memory is independent of subspace size, fit them to minimize gradient estimation bias, and pipeline per-layer gradient offload, CPU update and parameter upload with the backward pass.
Key results
- On instruction tuning, LSP-Offload reduces end-to-end fine-tuning time by 33.1%-62.5% over Zero-Offload when converging to the same accuracy.
- On GLUE with RoBERTa-base, LSP (d=512, r=16) reaches 0.855 average accuracy vs 0.836 for full parameter tuning and 0.844 for GaLore, using 253MB vs 747MB GPU memory.
- On HumanEval with Deepseek-Coder-1.3B, LSP (d=1280, r=4) averages 45.6 vs 39.3 for LoRA and 36.4 for GaLore, at 3.6GB GPU memory on a 4GB laptop GPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learned sparse projectors instead of full-parameter tuning | Fine-tuning RoBERTa-base · GLUE Avg. (1 hour) | accuracy: 0.8362625 -> 0.855275 | Helped |
| Use learned sparse projectors instead of GaLore at Rank=16 | Fine-tuning RoBERTa-base · GLUE Avg. (1 hour) | accuracy: 0.84375 -> 0.855275 | Helped |
| Use LSP-Offload instead of Zero-Offload | Alpaca instruction fine-tuning · Alpaca, laptop GPU | time to same evaluation accuracy: 62.5% less time | Helped |
| Use LSP-Offload instead of Zero-Offload | Alpaca instruction fine-tuning · Alpaca, workstation GPU | time to same evaluation accuracy: 33.1% less time | Helped |
| Use LSP-Offload instead of LoRA | Alpaca instruction fine-tuning · Alpaca | evaluation perplexity: 30% lower evaluation perplexity | Helped |
| Use LSP-Offload instead of GaLore | DeepSeek-Coder-1.3B fine-tuning · HumanEval | average accuracy: 27.8% higher average accuracy | Helped |
| Use LSP-Offload instead of Zero-Offload | DeepSeek-Coder-6.7B fine-tuning · HumanEval, 15h | average accuracy: 2.4% | Helped |
| Add layer-wise scheduling (ablation) | Zero-Offload · Training throughput | iterations per unit time: 18% | Helped |
| Use LSP-Offload with subspace size 256 instead of native training | LSP-Offload · Training throughput | throughput: 10.6% slowdown | Hurt |
| Use LSP-Offload with subspace size 512 instead of native training | LSP-Offload · Training throughput | throughput: 16.7% | Hurt |
| Increase subspace size d (ablation) | Learned sparse projectors · DeepSeek-Coder coding task | estimation bias | Helped |
| Increase nonzero entries per row r (ablation) | Learned sparse projectors · DeepSeek-Coder fine-tuning | generalization | Hurt |
Surprising findings
- Smaller sparse projector nonzeros r (4 or 8) generalize better than larger r, contrary to expectation that more nonzeros help.
- GaLore achieves lower training estimation error than sparse projectors at r>=16, yet sparse projectors generalize better on test estimation error.
Limitations
- Introduces several new hyperparameters (d, r, check frequency, update threshold) that affect performance.
- Compression and decompression add GPU compute overhead; sparse kernels not yet implemented.
- DeepSeek-Coder experiments are simulated rather than run with actual offloading.
Experimental setup
Datasets: GLUE Alpaca WizardCoder-style code instruction dataset HumanEval
Benchmarks: GLUE Alpaca HumanEval
Metrics: accuracy evaluation perplexity training loss end-to-end training time GPU memory training throughput estimation bias
Baselines: ZeRO-Offload LoRA GaLore Full parameter fine-tuning
Builds on: ZeRO-Offload GaLore LoRA
Closest papers by meaning
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.90)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.90)
- Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs (ICML 2025 · similarity 0.90)
- 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11) (NeurIPS 2025 · similarity 0.90)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.90)
- ConServe: Fine-Grained GPU Harvesting for LLM Online and Offline Co-Serving (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.