Research map / Architectures and efficiency / Pruning and efficient tuning
LoQT: Low-Rank Adapters for Quantized Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoQT, which trains low-rank factors on top of NF4-quantized full-rank weights (GaLore-style gradient-projected P, trainable B, periodic merging on an exponentially growing schedule) to enable memory-efficient quantized pretraining and fine-tuning on a 24GB GPU.
Problem
Pretraining large models on consumer hardware is infeasible without sharding, offloading, or per-layer gradient updates, and low-rank adapters like LoRA do not work well for pretraining from scratch; updating quantized weights directly is also hard because small gradient updates vanish in low precis
Approach
Initialize a fixed projection P from the SVD of the gradient (GaLore-style) and a trainable B to minimize quantization error, train only B, periodically merge PB into the NF4-quantized W with error compensation, and use an exponentially increasing merge interval so accumulated updates survive quantization.
Key results
- On C4 pretraining, LoQT reaches 15.55 perplexity at 1B (vs 15.56 full-rank) using 3.16G memory versus 7.80G for full training.
- On GLUE with DeBERTaV3-base at rank 32, LoQT averages 88.7 versus 88.5 for GaLore, 88.4 for LoftQ and 88.3 for LoRA.
- Error compensation improves perplexity by about 3.5 points and exponentially increasing update intervals by a further 1.5 points on the 130M model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoQT instead of full-rank training | LLaMA2-style pretraining · C4, 60M | perplexity: 33.32 $\pm$ 0.22 -> 33.98 $\pm$ 0.15 | Hurt |
| Use LoQT instead of full-rank training | LLaMA2-style pretraining · C4, 130M | perplexity: 24.51 $\pm$ 0.03 -> 24.57 $\pm$ 0.01 | No clear effect |
| Use LoQT instead of full-rank training | LLaMA2-style pretraining · C4, 1B | perplexity: 15.56 -> 15.55 | No clear effect |
| Quantize the model weights (LoQT versus PLoRA) | LoQT pretraining · C4, 350M | perplexity: 18.85 $\pm$ 0.01 -> 19.12 $\pm$ 0.01 | Hurt |
| Use LoQT instead of LoRA | DeBERTaV3-base fine-tuning · GLUE development average | average score: 88.3 -> 88.7 | Helped |
| Remove quantization (ablation) | LoQT fine-tuning · GLUE development average | average score: 88.7 -> 89.1 | Helped |
| Omit merging updates into pretrained weights (ablation) | LoQT fine-tuning · GLUE development average | average score: 88.7 -> 88.9 | Hurt |
| Add quantization error compensation (ablation) | LoQT pretraining · C4, 130M | perplexity: approximately 3.5 points better perplexity | Helped |
| Use exponentially increasing update intervals (ablation) | LoQT pretraining · C4, 130M | perplexity: additional 1.5 points | Helped |
| Quantize P as well as W (ablation) | LoQT pretraining · C4, 130M | validation loss | Hurt |
| Use exponentially increasing rather than fixed projection gaps (ablation) | LoQT and GaLore · C4 pretraining | perplexity | Helped |
| Reduce adapter rank to 64 (ablation) | LoQT · C4, 130M | perplexity | Hurt |
Surprising findings
- Quantized LoQT attains results close to non-quantized PLoRA and sometimes surpasses GaLore, LoftQ and LoRA on GLUE fine-tuning.
- The exponentially increasing projection gap helps quantized models match non-quantized performance, and also gives modest gains for GaLore without quantization.
Limitations
- Pretraining was not run to convergence for models above 1B due to compute limits.
- Quantized variant shows a slight perplexity drop relative to non-quantized PLoRA, and very low rank (64) diverges with quantization.
- Only weight-only quantization is implemented, so throughput is lower than standard AdamW training.
Experimental setup
Datasets: C4 GLUE GSM8K Icelandic text dataset (curated subset)
Benchmarks: C4 validation perplexity GLUE GSM8K Icelandic perplexity
Metrics: validation perplexity accuracy Matthews correlation F1 Pearson correlation memory (GB) tokens per second
Baselines: Full-rank training LoRA ReLoRA GaLore LoftQ QLoRA
Builds on: LoRA GaLore ReLoRA LoftQ QLoRA
Closest papers by meaning
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.89)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.89)
- GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models (ICML 2025 · similarity 0.89)
- GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (ICML 2024 · similarity 0.89)
- Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models (ICLR 2025 · similarity 0.89)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.