Research map / Large language models / Instruction and fine-tuning
VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VeLoRA, which splits activation tokens into sub-tokens, projects them onto a fixed rank-1 vector during the forward pass, and coarsely reconstructs them in the backward pass to cut activation memory for fine-tuning and pre-training.
Problem
Training and fine-tuning LLMs and transformers is memory-intensive, largely because intermediate activations must be stored for backpropagation.
Approach
Compress saved input activations by grouping tokens into sub-tokens and projecting each onto a single fixed unit vector initialised from the average of the first batch's sub-tokens, avoiding SVD and recomputation; the method is shown to be a special case of LoRA with data-driven initialisation.
Key results
- On VTAB-1k subset, VeLoRA on full tuning reaches 71.2 average vs 69.7 for full tuning, and with LoRA improves 72.1 to 72.7 while reducing memory 2.86 to 2.74 GB.
- On LLaMA-7B Alpaca fine-tuning, VeLoRA reaches 39.5 MMLU vs 39.0 for QLoRA with memory 4.88 vs 5.77 GB.
- On C4 pre-training with LLaMA-130M, VeLoRA reaches 25.29 perplexity vs 25.36 for GaLore with 1.83G vs 2.02G memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VeLoRA activation compression to full tuning | ViT-Base-224/16 · VTAB-1k average | accuracy: 69.7 -> 71.2 | Helped |
| Add VeLoRA activation compression to SSF | ViT-Base-224/16 · VTAB-1k average | accuracy: 72.7 -> 72.6 | Hurt |
| Add VeLoRA activation compression to Hydra | ViT-Base-224/16 · VTAB-1k average | accuracy: 74.7 -> 74.8 | Helped |
| Add VeLoRA activation compression to LoRA | ViT-Base-224/16 · VTAB-1k average | accuracy: 72.1 -> 72.7 | Helped |
| Use VeLoRA instead of full fine-tuning | RoBERTa-Base · GLUE average | accuracy: 86.28 -> 85.91 | Hurt |
| Use VeLoRA instead of QLoRA | LLaMA fine-tuning · MMLU test, 5-shot | accuracy: +0.5 pp | Helped |
| Use VeLoRA instead of GaLore | LLaMA pre-training · C4 validation (60M) | perplexity: 1.08 pp lower perplexity | Helped |
| Use fixed first-batch average initialization rather than random initialization (ablation) | VeLoRA on LLaMA-7B · MMLU | accuracy: 36.8 -> 39.5 | Helped |
| Use fixed first-batch average initialization rather than SVD initialization (ablation) | VeLoRA on LLaMA-7B · MMLU | accuracy: 37.1 -> 39.5 | Helped |
| Use fixed first-batch average initialization rather than running-average initialization (ablation) | VeLoRA on LLaMA-7B · MMLU | accuracy: 38.9 -> 39.5 | Helped |
| Set sub-token size to D / 8 rather than D / 32 (ablation) | VeLoRA on LLaMA-7B · MMLU | accuracy: 39.5 -> 37.2 | Hurt |
| Set sub-token size to D / 64 rather than D / 32 (ablation) | VeLoRA on LLaMA-7B · MMLU | accuracy: 39.5 -> 37.9 | Hurt |
Surprising findings
- Projecting sub-tokens onto a fixed rank-1 vector, which discards most gradient magnitude and direction, does not hurt convergence and can even improve accuracy.
- Initialising the projection vector via SVD gave lower accuracy (37.1) than a simple fixed average (39.5).
Limitations
- Evaluated only on Transformer models; CNNs, RNNs and SSMs are untested.
- Primary benefit is memory reduction; training time overhead remains.
- No error bars or statistical significance reported.
Experimental setup
Datasets: VTAB-1k GLUE Alpaca C4 MMLU
Benchmarks: VTAB-1k GLUE MMLU C4
Metrics: accuracy perplexity GPU memory (GB) iterations per second
Baselines: Full fine-tuning LoRA SSF Hydra GaLore QLoRA FLoRA LOMO gradient checkpointing
Builds on: LoRA GaLore FLoRA QLoRA gradient checkpointing
Closest papers by meaning
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.91)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.91)
- AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections (NeurIPS 2025 · similarity 0.91)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.90)
- AdaRankGrad: Adaptive Gradient Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning (ICLR 2025 · similarity 0.89)
- ReLoRA: High-Rank Training Through Low-Rank Updates (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.