Research map / Architectures and efficiency / Pruning and efficient tuning
From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes state-based fine-tuning (State-FT) that adds parallel low-rank control edges across whole residual blocks (Parallel/Double Control) instead of weight updates, reducing GPU memory and training time relative to LoRA and DoRA.
Problem
LoRA-style weight-centric PEFT still requires storing large intermediate forward states, so memory savings are limited; training 7B/8B models on consumer GPUs remains infeasible.
Approach
Reframe fine-tuning as perturbing forward states in the computational graph; add a parallel low-rank control edge across an entire residual block (FFN or attention+FFN) so intermediate states inside the block need not be stored, yielding a control-affine structure that generalizes LoRA.
Key results
- On ViT-B with CIFAR-100, Control uses 12.280 GB vs LoRA 18.010 GB at similar accuracy (91.96 vs 91.84) and 3h24m vs 4h42m training time.
- On LLaMA2-7B commonsense, DoubleControl uses 35.214 GB vs DoRA 59.568 GB with average accuracy 79.6 vs 79.7.
- Double control trains LLaMA2-7B on a single RTX 3090 (24GB) at 20.656 GB memory without quantization, with average accuracy 78.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use parallel control instead of LoRA | ViT-B · CIFAR-100 | GPU memory: 18.010 GB -> 12.280 GB | Helped |
| Use parallel control instead of LoRA | ViT-B · CIFAR-100 | training time: 4h 42m -> 3h 24m | Helped |
| Use parallel control instead of LoRA | ViT-B · CIFAR-100 | accuracy: 91.84˘0.11 -> 91.96˘0.05 | No clear effect |
| Use parallel control on MLP layers instead of LoRA | RoBERTa-base · GLUE | average score: 85.62 -> 86.14 | Helped |
| Use parallel control on MLP layers instead of LoRA | RoBERTa-large · GLUE | average score: 88.69 -> 89.28 | Helped |
| Use parallel control instead of LoRA | RoBERTa-base · CoLA | GPU memory: 14.030 GB -> 12.634 GB | Helped |
| Use parallel control instead of LoRA | RoBERTa-base · CoLA | training time: 5m 33s -> 4m 59s | Helped |
| Combine control on U and D with LoRA on Q, K, and V instead of DoRA | LLaMA2-7B · Commonsense Benchmark | average score: 79.7 -> 80.0 | Helped |
| Use double control on Q, K, V, U, and D instead of DoRA | LLaMA2-7B · Commonsense Benchmark | GPU memory: 59.568 GB -> 35.214 GB | Helped |
| Use double control on Q, K, V, U, and D instead of DoRA | LLaMA2-7B · Commonsense Benchmark | average score: 79.7 -> 79.6 | No clear effect |
| Use double control on Q, K, V, U, and D instead of DoRA | LLaMA3-8B · Commonsense Benchmark | GPU memory: 67.284 GB -> 45.316 GB | Helped |
| Control the attention layer individually rather than the MLP layer (ablation) | ViT · ViT tuning granularity | performance: 91.96˘0.05 -> 91.93˘0.03 | Hurt |
Surprising findings
- Halving DoRA's rank cuts parameters ~50% but reduces GPU memory only 0.256 GB, indicating intermediate states rather than parameter count dominate memory.
Limitations
- Only edges tied to residual connections are explored; the broader state-based framework is not fully exploited.
- Control-affine theoretical analysis is left for future work.
- Block-level control underperforms slightly on ViT (91.53 vs 91.96) relative to MLP-only control.
Experimental setup
Datasets: CIFAR-100 GLUE Commonsense Benchmark (BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA)
Benchmarks: CIFAR-100 GLUE Commonsense Benchmark
Metrics: accuracy Matthews correlation Pearson correlation GPU memory training time number of parameters
Baselines: LoRA DoRA VeRA ChatGPT
Builds on: LoRA DoRA VeRA control-theoretic fine-tuning (Zhang et al., 2024b)
Closest papers by meaning
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.87)
- Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.87)
- Block-Diagonal LoRA for Eliminating Communication Overhead in Tensor Parallel LoRA Serving (NeurIPS 2025 · similarity 0.86)
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.86)
- SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning (ICML 2025 · similarity 0.85)
- LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.