Research map / Large language models / Instruction and fine-tuning
LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoftQ, which alternates quantization and SVD-based low-rank approximation to initialize LoRA adapters for quantized pre-trained LLMs and encoder/encoder-decoder models, closing the gap to full fine-tuning especially at 2-bit.
Problem
Quantizing a pre-trained model and then applying LoRA fine-tuning yields a gap to full fine-tuning; QLoRA's zero-initialized adapters on a quantized backbone start from a discrepant point, which degrades or breaks fine-tuning at low bit-widths (below 3 bits).
Approach
Jointly approximate the pre-trained weight by a quantized backbone plus low-rank adapters via alternating optimization: quantize the residual W minus the current low-rank term, then set the adapters to the rank-r SVD of the new quantization residual; the result initializes LoRA fine-tuning on the frozen quantized backbone.
Key results
- On DeBERTaV3-base with 2-bit uniform quantization at rank 32, LoftQ reaches 88.0% MNLI-m versus 79.9% for QLoRA.
- On LLAMA-2-7b with 2-bit NF2, QLoRA fails to converge while LoftQ reaches 7.85 WikiText-2 perplexity and 20.9% GSM8K accuracy.
- On BART-large 4-bit NF4 (rank 16), LoftQ improves XSum ROUGE-1 from 43.29 to 44.51 over QLoRA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LoftQ initialization instead of QLoRA initialization | DeBERTaV3-base, 2-bit NF2, rank 16 · MNLI development | accuracy (m / mm): 75.4/75.6 -> 84.7/85.1 | Helped |
| Use LoftQ initialization instead of QLoRA initialization | DeBERTaV3-base, 2-bit uniform, rank 32 · MNLI development | accuracy (m / mm): 79.9/79.5 -> 88.0/88.1 | Helped |
| Use LoftQ initialization instead of QLoRA initialization | BART-large, 4-bit NF4, rank 8 · XSum test | ROUGE-1/2/L: 42.91/19.72/34.82 -> 44.08/20.72/35.89 | Helped |
| Use LoftQ initialization instead of QLoRA initialization | BART-large, 4-bit NF4, rank 8 · CNN/DailyMail test | ROUGE-1/2/L: 43.10/20.22/40.06 -> 43.81/20.95/40.84 | Helped |
| Use LoftQ initialization instead of QLoRA initialization | BART-large, 2-bit NF2, rank 8 · XSum test | ROUGE-1/2/L | Helped |
| Use LoftQ initialization instead of QLoRA initialization | LLAMA-2-7b, 2-bit NF2 · WikiText-2 | perplexity | Helped |
| Use LoftQ initialization instead of QLoRA initialization | LLAMA-2-13b, 4-bit NF4 · GSM8K | accuracy: 39.9 -> 45.0 | Helped |
| Use mixed precision with 4-bit quantization in the first 4 layers and 2-bit quantization in the rest | LoftQ on LLAMA-2-13b · GSM8K | accuracy | Helped |
| Perform SVD before quantization in each alternating step rather than quantization before SVD (ablation) | LoftQ, DeBERTaV3-base, rank 32 · MNLI development, 2-bit uniform | accuracy (m / mm): 88.0/88.1 -> 87.8/87.7 | Hurt |
| Add weight decay to full-precision LoRA fine-tuning (ablation) | LLAMA-2-13b · GSM8K | accuracy: 43.1 -> 45.3 | Helped |
| Add weight decay to full-precision LoRA fine-tuning (ablation) | LLAMA-2-7b · GSM8K | accuracy: 36.9 -> 34.4 | Hurt |
Surprising findings
- LoftQ at 4-bit on LLAMA-2-13b GSM8K (45.0) outperforms full-precision LoRA (43.1), which the authors attribute to implicit regularization from quantization.
- QLoRA's LoRA initialization fails to converge on several tasks (e.g., CoLA, ANLI at 2-bit) while LoftQ converges.
Limitations
- Gains in 4-bit settings are small relative to QLoRA on many tasks, with most benefit concentrated at 2-bit.
- Quantization still introduces residual error; increasing alternating steps yields diminishing returns.
- Evaluated on relatively small encoder/encoder-decoder models and LLAMA-2 at 7B/13B only.
Experimental setup
Datasets: MNLI QNLI RTE SST-2 MRPC CoLA QQP STS-B SQuAD v1.1 ANLI XSum CNN/DailyMail
Benchmarks: GLUE SQuAD v1.1 ANLI XSum CNN/DailyMail GSM8K WikiText-2
Metrics: accuracy Matthews correlation exact match / F1 ROUGE-1/2/L perplexity
Baselines: Full fine-tuning LoRA QLoRA LoSparse
Builds on: QLoRA LoRA NormalFloat quantization HuggingFace Transformers
Closest papers by meaning
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.93)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.92)
- Loquetier: A Virtualized Multi-LoRA Framework for Unified LLM Fine-tuning and Serving (NeurIPS 2025 · similarity 0.90)
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.90)
- Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning (NeurIPS 2026 · similarity 0.90)
- RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.