Research map / Architectures and efficiency / Quantization
RILQ: Rank-Insensitive LoRA-Based Quantization Error Compensation for Boosting 2-Bit Large Language Model Accuracy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RILQ, a LoRA-based quantization error compensation method that uses a model-level activation discrepancy loss (plus ground-truth LM loss) to initialize adapters, making 2-bit LLM accuracy robust to low ranks.
Problem
LoRA-based quantization error compensation (LQEC) underperforms at sub-4-bit (especially 2-bit) weight quantization, and the reason for this limitation has not been understood; 2-bit quantization errors are high-rank, conflicting with LoRA's low-rank assumption.
Approach
Rank sensitivity analysis shows that widening the discrepancy scope from linear modules to the full model (loss on final Transformer layer output) makes LoRA adapters rank-insensitive; RILQ optimizes all adapters jointly with this model-level loss plus a causal LM ground-truth loss, letting rank-critical and rank-redundant modules compensate cooperatively.
Key results
- On 2-bit LLaMA-3-8B with QuIP#, RILQ improves average zero-shot CSQA accuracy by about 8.1%.
- With 2-bit OmniQuant on LLaMA-2-7B, RILQ at rank 16 achieves C4 perplexity 10.79 versus 12.71 for SVD at rank 16, beating SVD even at rank 256 (10.99).
- LoftQ at 2-bit on LLaMA-2-7B drops average zero-shot accuracy to 37.06 (vs 64.80 baseline), while RILQ raises it to 49.54.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply RILQ compensation to QuIP# quantization | LLaMA-2-7B · W2A16 zero-shot CSQA Avg. | accuracy: 54.38 -> 60.12 | Helped |
| Apply RILQ compensation to OmniQuant quantization | LLaMA-2-7B · W2A16 zero-shot CSQA Avg. | accuracy: 51.88 -> 55.52 | Helped |
| Apply RILQ compensation to QuaRot quantization | LLaMA-3-8B · W2A16 zero-shot CSQA Avg. | accuracy: 49.02 -> 58.43 | Helped |
| Initialize task-specific fine-tuning adapters with RILQ | LLaMA-2-7B with OmniQuant · PIQA fine-tuning | accuracy: 77.31 -> 77.91 | Helped |
| Initialize task-specific fine-tuning adapters with RILQ | LLaMA-2-7B with OmniQuant · GSM8K fine-tuning | accuracy: 28.66 -> 30.86 | Helped |
| Add RILQ compensation in QA-LoRA | LLaMA-2-7B · W2A16 CSQA | accuracy: 47.42 -> 54.51 | Helped |
| Use model-level activation discrepancy loss instead of linear-level discrepancy loss (ablation) | LLaMA-2-7B quantization error compensation · Zero-shot CSQA Avg. | accuracy: 45.02 -> 52.77 | Helped |
| Add GT-Loss to Model-Loss (ablation) | LLaMA-2-7B quantization error compensation · Zero-shot CSQA Avg. | accuracy: 52.77 -> 54.71 | Helped |
| Use GT-Loss only instead of Model-Loss plus GT-Loss (ablation) | LLaMA-2-7B quantization error compensation · Zero-shot CSQA Avg. | accuracy: 53.28 -> 54.71 | Hurt |
| Increase RILQ adapter rank from 16 to 256 (ablation) | LLaMA-2-7B with OmniQuant · W2A16 C4 | perplexity: 10.79 -> 10.72 | No clear effect |
| Optimize Model-Loss against logits instead of the final layer's output activation (ablation) | LLaMA-2-7B with OmniQuant · 2-bit WikiText2 | perplexity: 9.18 -> 9.28 | Hurt |
| Add RILQ to QuIP# end-to-end fine-tuning (ablation) | LLaMA-3-8B · W2A16 CSQA Avg. | accuracy: 59.3 -> 61.3 | Helped |
Surprising findings
- Model-Loss yields lower relative error at the LM head despite higher relative error in intermediate layers, suggesting internal activation drift helps final alignment.
- Using the final-layer hidden state instead of logits for Model-Loss does not significantly hurt performance (C4 PPL 10.70 vs 10.88).
Limitations
- Longer calibration sequences or more samples improve perplexity but increase time to over 7 hours.
- Gains are smaller for 3-bit quantization where baselines already approach FP16 accuracy.
- Evaluation limited to LLaMA-2 and LLaMA-3 families and a few quantizers and tasks.
Experimental setup
Datasets: C4 WikiText-2 PIQA ARC-Challenge ARC-Easy GSM8K WinoGrande HellaSwag
Benchmarks: WinoGrande PIQA HellaSwag ARC-Challenge ARC-Easy GSM8K WikiText-2 C4
Metrics: accuracy perplexity average accuracy
Baselines: LoftQ OmniQuant QuIP# QuaRot QA-LoRA RA-LoRA QuIP#-FT SVD initialization
Builds on: LoRA LoftQ ApiQ QA-LoRA OmniQuant
Closest papers by meaning
- LQER: Low-Rank Quantization Error Reconstruction for LLMs (ICML 2024 · similarity 0.91)
- LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning (ICLR 2024 · similarity 0.90)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.90)
- QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.90)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.90)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.