Research map / Architectures and efficiency / Pruning and efficient tuning
ReLoRA: High-Rank Training Through Low-Rank Updates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReLoRA, which trains high-rank transformer language models via a sequence of merged low-rank LoRA updates with optimizer pruning, jagged LR restarts, and a full-rank warm start, reaching near full-rank perplexity up to 1.3B parameters.
Problem
Training large overparameterized transformers from scratch is extremely costly, and plain low-rank (LoRA) training yields poor high-rank updates, so it is unclear whether parameter-efficient methods can be used for pre-training.
Approach
Periodically merge LoRA factors into the frozen weights and reinitialize them, so that the sum of successive low-rank updates forms a high-rank update; stabilize restarts with partial (99%) optimizer-state magnitude pruning, a jagged cosine LR schedule with re-warmup, and a short full-rank warm start.
Key results
- At 1.3B, ReLoRA reaches perplexity 17.27 vs 16.83 for full-rank training and 18.23 for LoRA + warm start (30K steps, 23.1B tokens).
- On 8xA100, ReLoRA gives a 9% total wall-clock speedup for 1.3B training; on 2x3090 the total speedup is 51%.
- ReLoRA reduces GPU RAM use by 5.5GB per GPU at the same microbatch size (27.8GB to 22.3GB).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace LoRA + Warm Start with ReLoRA | C4 language model pretraining · C4, 1.3B @30K steps | perplexity: 18.23 -> 17.27 | Helped |
| Replace ReLoRA with Online ReLoRA, merging LoRA parameters more frequently | ReLoRA · C4, 250M (@15k steps) | perplexity: 27.66 -> 29.31 | Hurt |
| Increase ReLoRA rank from 128 to 512 (ablation) | ReLoRA · C4, 1.3B, 10K iterations after 5K-step warm start | perplexity: 19.16 -> 19.00 | No clear effect |
| Add restarts to LoRA without optimizer reset, jagged schedule, or warm start (ablation) | ReLoRA, C4 130M · C4, 130M | perplexity: 34.17 -> 34.25 | No clear effect |
| Add optimizer reset along with restarts, without jagged schedule (ablation) | ReLoRA, C4 130M · C4, 130M | perplexity: (diverged) | Hurt |
| Add a jagged schedule to restarts and optimizer reset (ablation) | ReLoRA, C4 130M without warm start · C4, 130M | perplexity: (diverged) -> 29.77 | Helped |
| Add warm start to ReLoRA with restarts, optimizer reset, and jagged schedule (ablation) | ReLoRA, C4 130M · C4, 130M | perplexity: 29.77 -> 25.04 | Helped |
| Vary LoRA rank from 16 to 64 (ablation) | ReLoRA fine-tuning on T5-large · QNLI | accuracy: 94.05 -> 94.55 | Helped |
| Increase optimizer-state pruning from 85% to 99% (ablation) | ReLoRA fine-tuning on T5-large · QNLI | accuracy: 94.51 -> 94.56 | No clear effect |
| Change reset schedule from 6000 iterations (1 reset) to 4000 iterations (2 resets) (ablation) | ReLoRA fine-tuning on T5-large · QNLI | accuracy: 94.38 -> 94.73 | Helped |
Surprising findings
- Naive optimizer reset with ReLoRA causes divergence; a jagged LR schedule is required for stability.
- Online ReLoRA (merging every 100 iterations) performs worse than regular ReLoRA at both 250M and 1.3B scales.
- ReLoRA fine-tuning on GLUE does not outperform LoRA, despite strong pre-training results.
Limitations
- Speedup is strongly hardware-dependent and small (9%) on high-bandwidth 8xA100 setups.
- Requires a full-rank warm start (about 1/4 to 1/3 of training) and is not fully parameter-efficient from scratch.
- Evaluated only up to 1.3B parameters; ReLoRA fine-tuning does not beat LoRA on GLUE.
Experimental setup
Datasets: C4 GLUE
Benchmarks: C4 validation perplexity GLUE
Metrics: perplexity accuracy Matthews correlation wall-clock training time throughput GPU RAM
Baselines: Full-rank training LoRA LoRA + Warm Start Control (low-rank-size full-rank model) Online ReLoRA
Builds on: LoRA Adam Lottery Ticket Hypothesis Chinchilla scaling laws
Closest papers by meaning
- VeLoRA: Memory Efficient Training using Rank-1 Sub-Token Projections (NeurIPS 2024 · similarity 0.89)
- RandLoRA: Full rank parameter-efficient fine-tuning of large models (ICLR 2025 · similarity 0.89)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.87)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.86)
- GeoLoRA: Geometric integration for parameter efficient fine-tuning (ICLR 2025 · similarity 0.86)
- Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.