Research map / Large language models / Instruction and fine-tuning
Gradient Weight-normalized Low-rank Projection for Efficient LLM Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GradNormLoRP, which column-normalizes weights, trains a magnitude vector plus low-rank direction updates, and projects low-rank gradients via periodically updated SVD projections to cut optimizer memory for LLM pre-training and fine-tuning.
Problem
Low-rank PEFT methods such as LoRA often underperform full fine-tuning and still incur high activation and optimizer memory, making LLM training on consumer GPUs difficult.
Approach
Combine weight vector normalization (each column = trainable magnitude times unit direction), a LoRA-style low-rank update to the direction matrix, and GaLore-style gradient low-rank projection refreshed every 250 steps, so optimizer states are kept in a compact low-rank subspace.
Key results
- On GLUE with RoBERTa-base at rank 8, GradNormLoRP averages 80.65 versus LoRA's 79.23 and full fine-tuning's 80.32.
- On LLaMA 1B pre-training on C4, GradNormLoRP reaches perplexity 16.12 with 3.81G memory versus GaLore's 16.77 and 4.69G.
- 8-bit GradNormLoRP pre-trains LLaMA 7B in 15.29GB memory (vs 18.36GB for 8-bit GaLore and 26.47GB for 8-bit Adam) and reaches perplexity 15.41 at 80K steps versus 16.08 for 8-bit GaLore.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GradNormLoRP instead of LoRA at rank 8 | RoBERTa-base fine-tuning · GLUE | average score: 79.23 -> 80.65 | Helped |
| Use GradNormLoRP instead of full fine-tuning | RoBERTa-base fine-tuning · GLUE | average score: 80.32 -> 80.65 | Helped |
| Use GradNormLoRP instead of GaLore | LLaMA 350M pre-training · C4 | validation perplexity: 20.64 -> 19.28 | Helped |
| Use GradNormLoRP instead of GaLore | LLaMA 1B pre-training · C4 | memory: 4.69G -> 3.81G | Helped |
| Use 8-bit GradNormLoRP instead of 8-bit Adam | LLaMA 7B pre-training · C4, 80K steps | validation perplexity: 16.24 -> 15.41 | Helped |
| Use 8-bit GradNormLoRP instead of 8-bit Adam | LLaMA 7B pre-training · C4, 80K steps | memory: 26.47G -> 15.29G | Helped |
| Use rank 4 rather than rank 8 (ablation) | GradNormLoRP on RoBERTa-base · GLUE | average score: 80.65 -> 81.01 | Helped |
| Use a moderate subspace update frequency of around 250 iterations rather than overly frequent or infrequent updates (ablation) | GradNormLoRP pre-training · C4 | validation perplexity | Helped |
| Train rank 128 for 80K steps rather than rank 512 for 20K steps (ablation) | GradNormLoRP pre-training · C4 | validation perplexity | Helped |
| Add weight normalization (ablation) | GradNormLoRP on RoBERTa-base · GLUE | task scores | Helped |
| Add gradient projection (ablation) | GradNormLoRP on RoBERTa-base · GLUE | task scores | Helped |
| Use RoBERTa-large instead of RoBERTa-base (ablation) | GradNormLoRP fine-tuning · GLUE | average score: 80.63 -> 82.43 | Helped |
Surprising findings
- Reported GLUE gains over full fine-tuning on several models (e.g., RoBERTa-base average 80.65 vs 80.32), which the authors attribute to normalization improving gradient conditioning.
Limitations
- Large-scale 7B experiment uses a single trial without hyperparameter tuning and compares only against 8-bit Adam and 8-bit GaLore.
- Theoretical lemma assumes simplified gradient forms (quadratic-like objectives, PSD matrices) and does not directly cover LLM training dynamics.
- Some GLUE results are mixed; e.g., WNLI and some tasks show GradNormLoRP below LoRA or GaLore.
Experimental setup
Datasets: GLUE C4
Benchmarks: GLUE C4
Metrics: accuracy Matthews correlation Pearson/Spearman correlation validation perplexity memory usage
Baselines: Full fine-tuning LoRA DoRA GaLore 8-bit Adam 8-bit GaLore
Builds on: LoRA GaLore DoRA weight normalization
Closest papers by meaning
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.92)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.91)
- Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures (ICLR 2025 · similarity 0.91)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.90)
- Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer (ICLR 2025 · similarity 0.89)
- Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.