Research map / Large language models / Instruction and fine-tuning
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QA-LoRA, which uses group-wise pooling of inputs before LoRA adapters so that LLM weights can be quantized (INT4/3/2) during fine-tuning and the merged adapter keeps the model in low-bit form without post-training quantization loss.
Problem
Fine-tuning LLMs with LoRA is memory-heavy, and naively combining PEFT with quantization either loses accuracy after post-training quantization or breaks the quantized form after merging adapters back into the weights.
Approach
Use group-wise quantization of pre-trained weights (scaling/zero factors per group of input rows) and pool the input features within each group before the LoRA matrices, so the low-rank adapter can be merged into the zero-point matrix while keeping the weights INT-quantized, balancing the degrees of freedom of quantization and adaptation.
Key results
- On LLaMA-7B with FLAN v2, 4-bit QA-LoRA reaches 45.9% 0-shot MMLU average versus 41.6% for QLoRA w/ GPTQ at 4-bit.
- At 2 bits on LLaMA-7B (Alpaca), QA-LoRA attains 26.5% 0-shot MMLU, whereas QLoRA w/ GPTQ gets 23.2%; on LLaMA-65B 2-bit commonsense QA improves by about 15% over post-quantized QLoRA on LLaMA-7B.
- QA-LoRA cuts LLaMA-7B fine-tuning time from 40.0h to 21.5h relative to QLoRA on one V100 with 10K steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QA-LoRA instead of QLoRA with GPTQ for INT4 | LLaMA-7B fine-tuned on Alpaca · MMLU 5-shot | accuracy (Avg.): 36.0 -> 39.4 | Helped |
| Use QA-LoRA instead of QLoRA with GPTQ for INT2 | LLaMA-7B fine-tuned on Alpaca · MMLU 0-shot | accuracy (Avg.): 23.2 -> 26.5 | Helped |
| Use QA-LoRA instead of QLoRA with GPTQ | LLaMA-7B · Commonsense QA 0-shot | accuracy (Avg.): 59.8 -> 61.8 | Helped |
| Use QA-LoRA instead of QLoRA | LLaMA-7B Alpaca fine-tuning · fine-tuning | time (h): 40.0 -> 21.5 | Helped |
| Use FLAN v2 rather than Alpaca for fine-tuning | QA-LoRA on LLaMA2-7B, INT4 · MMLU 0-shot | accuracy (Avg.): 43.9 -> 50.5 | Helped |
| Use group size 32 rather than group size 128 (ablation) | QA-LoRA on LLaMA-7B, INT4 · MMLU 0-shot | accuracy (Avg.): 38.0 -> 39.4 | Helped |
| Use group size 32 rather than group size 128 (ablation) | QA-LoRA on LLaMA-7B, INT2 · MMLU 0-shot | accuracy (Avg.): 25.0 -> 26.5 | Helped |
| Use group size 32 rather than group size 128 (ablation) | QA-LoRA on LLaMA-7B, INT2 · MMLU 5-shot | accuracy (Avg.): 26.7 -> 27.5 | Helped |
| Fine-tune LLaMA2-7B with Alpaca rather than its original FP16 model | QA-LoRA, INT4 · MMLU 0-shot | accuracy (Avg.): 40.7 -> 43.9 | Helped |
| Fine-tune with QA-LoRA on the smaller Self-instruct, Longform, or Chip2 dataset rather than larger Alpaca/FLAN v2 datasets | LLaMA-7B · MMLU 5-shot | accuracy | Hurt |
Surprising findings
- The INT4 QA-LoRA version sometimes outperforms the original FP16-inference QLoRA (e.g., LLaMA-65B FLAN v2 0-shot 65.1 vs 62.8).
Limitations
- Gains are shown mainly on MMLU and commonsense QA with LLaMA families; generalization to other tasks and architectures is limited.
- Group-size-dependent storage and accuracy trade-offs; 2-bit results remain well below FP16 baselines on several settings.
- Relies on GPTQ as the post-training quantizer for the base weights.
Experimental setup
Datasets: Alpaca FLAN v2 Self-instruct Longform Chip2
Benchmarks: MMLU HellaSwag PIQA WinoGrande ARC-e ARC-c BoolQ OpenBookQA
Metrics: accuracy (0-shot) accuracy (5-shot) training time (h) number of learnable parameters
Baselines: QLoRA QLoRA w/ GPTQ PEQA LLaMA base models
Builds on: LoRA QLoRA GPTQ PEQA
Closest papers by meaning
- LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models (ICLR 2024 · similarity 0.93)
- C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Models (NeurIPS 2025 · similarity 0.92)
- GoRA: Gradient-driven Adaptive Low Rank Adaptation (NeurIPS 2025 · similarity 0.91)
- LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Uni-LoRA: One Vector is All You Need (NeurIPS 2025 · similarity 0.91)
- Bayesian Low-rank Adaptation for Large Language Models (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.