Research map / Large language models / Instruction and fine-tuning
Accurate LoRA-Finetuning Quantization of LLMs via Information Retention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IR-QLoRA, which improves LoRA-finetuned quantized LLMs via Information Calibration Quantization (entropy-maximizing calibration constant) and Information Elastic Connection (parameter-free connections in LoRA), evaluated on LLaMA and LLaMA2 with MMLU and CommonsenseQA.
Problem
LoRA-finetuning quantization of LLMs loses substantial information during quantization, and LoRA cannot effectively recover it, so low-bit quantized models degrade and fail to benefit from finetuning, especially at ultra-low bit-widths and large scales.
Approach
Treats quantization as information retention: ICQ searches a per-block calibration constant that maximizes entropy of quantized weights, and IEC adds parameter-free grouped-average and repeat connections around the LoRA matrices so LoRA can access original input features and diversify its transformation.
Key results
- 4-bit LLaMA-7B with IR-QLoRA reaches 40.8% 5-shot MMLU on Alpaca vs 38.4% for QLoRA and 39.4% for QA-LoRA.
- 2-bit IR-QLoRA on LLaMA-7B (Flan v2) reaches 33.7% MMLU vs 33.5% for QLoRA and 33.2% for QA-LoRA, a small margin over baselines.
- Additional fine-tuning time from ICQ search is only up to 0.84% of original fine-tuning time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IR-QLoRA instead of QA-LoRA | LLaMA2-7B · MMLU, Alpaca, 4-bit | accuracy: 43.9 -> 46.2 | Helped |
| Apply IR-QLoRA instead of QLoRA | LLaMA-7B · CommonsenseQA average, 4-bit | accuracy: 62.0 -> 63.7 | Helped |
| Add ICQ to Vanilla QLoRA (ablation) | LLaMA-7B · MMLU, 4-bit | accuracy: 38.4 -> 40.3 | Helped |
| Add IEC to Vanilla QLoRA (ablation) | LLaMA-7B · MMLU, 4-bit | accuracy: 38.4 -> 40.2 | Helped |
| Add IEC to the first LoRA sub-unit (ablation) | LLaMA-7B Vanilla QLoRA · MMLU, 4-bit | accuracy: 38.4 -> 39.4 | Helped |
| Add IEC to the latter LoRA sub-unit (ablation) | LLaMA-7B Vanilla QLoRA · MMLU, 4-bit | accuracy: 38.4 -> 39.7 | Helped |
| Add ICQ and IEC together (ablation) | LLaMA-7B · MMLU, 4-bit | accuracy: 38.4 -> 40.8 | Helped |
| Use ICQ instead of NormalFloat quantization (ablation) | LLaMA-7B without LoRA or fine-tuning · MMLU, 4-bit | accuracy: 35.1 -> 35.6 | Helped |
| Add ICQ and IEC to QA-LoRA | LLaMA-7B · MMLU, 4-bit | accuracy: 39.4 -> 39.9 | Helped |
| Apply IR-QLoRA instead of QA-LoRA | LLaMA2-13B · MMLU, Flan v2, 4-bit | accuracy: 56.6 -> 56.5 | Hurt |
Surprising findings
- 4-bit LLaMA-30B with finetuned QLoRA (57.7%) fails to reach the unfinetuned FP16 accuracy (58.2%) on MMLU.
- On LLaMA-65B Alpaca, IR-QLoRA ties QLoRA at 63.1% average MMLU accuracy.
Limitations
- Gains on large models are small (e.g., LLaMA-65B Alpaca average tied with QLoRA; LLaMA-30B +0.1 over QLoRA w/ GPTQ), and some sub-metrics regress.
- Ultra-low-bit gains over QLoRA at 2-bit are modest (e.g., 33.7 vs 33.5 on Flan v2) and far below full precision at 2 bits on Alpaca.
Experimental setup
Datasets: Alpaca Flan v2
Benchmarks: MMLU HellaSwag PIQA WinoGrande ARC-e ARC-c BoolQ OpenBookQA
Metrics: accuracy
Baselines: QLoRA QLoRA w/ GPTQ QA-LoRA PEQA NormalFloat LoftQ
Builds on: QLoRA QA-LoRA GPTQ LoRA NormalFloat quantization
Closest papers by meaning
- LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.88)
- Efficient Multi-task LLM Quantization and Serving for Multiple LoRA Adapters (NeurIPS 2024 · similarity 0.88)
- FALQON: Accelerating LoRA Fine-tuning with Low-Bit Floating-Point Arithmetic (NeurIPS 2025 · similarity 0.86)
- RILQ: Rank-Insensitive LoRA-Based Quantization Error Compensation for Boosting 2-Bit Large Language Model Accuracy (AAAI 2025 · similarity 0.85)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.85)
- ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.