Research map / Architectures and efficiency / Quantization

LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning

ICLR 2024 · Architectures and efficiency › Quantization · approach: Post-training quantization of LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LQ-LoRA, which decomposes each pretrained weight matrix into a fixed quantized component plus a trainable low-rank component, with ILP-based mixed-bit configuration and Fisher-weighted SVD initialization.

Problem

Memory-efficient finetuning and compression of LLMs: naive quantization plus LoRA with zero initialization of the low-rank adapter suffers at sub-4-bit quantization because quantization error is not accounted for, and uniform bit configs ignore that matrices differ in quantizability.

Approach

Alternating robust-PCA-style decomposition W ≈ Q + L1L2 (SVD for low-rank, NF quantization for Q), with an integer linear program assigning per-matrix quantization configurations under a global memory budget, and a Fisher-information-weighted SVD for data-aware initialization.

Key results

• LQ-LoRA (2.75-bit, rank 64, Fisher) on LLaMA-2-70B reaches C4 perplexity 6.35 vs 6.01 for 4-bit QLoRA and 6.42 for non-Fisher LQ-LoRA at 2.75 bits (Table 5/7 numbers).
• On RoBERTa-Large GLUE, Fisher LQ-LoRA at 2.5 bits scores 87.3 versus 75.4 for QLoRA at 2.5 bits and 88.5 for full finetuning.
• As a PTQ method, 2.75-bit LQ-LoRA (Fisher) on LLaMA-2-70B achieves WikiText perplexity 3.65 and C4 5.88, beating the listed 3-bit GPTQ and OmniQuant baselines on WikiText.

Measured effects reported in the paper

Surprising findings

• Fisher-weighted LQ-LoRA shows the largest advantage at 7B, while the gap shrinks at 70B.
• On the Open LLM benchmark, GSM8K dropped from 14.9 to 7.4 for the 7B model under 2.95-bit compression, indicating perplexity is not always commensurate with downstream performance.

Limitations

• The iterative decomposition algorithm is heuristic without convergence guarantees.
• Performance degrades significantly near 2.5 bits.
• Fisher-weighted variant requires backpropagation through the pretrained model to compute calibration Fisher matrices, which conflicts with the memory-efficient setting, although it is computed once.

Experimental setup

Datasets: C4 WikiText-2 OpenAssistant GLUE

Benchmarks: C4 perplexity WikiText-2 perplexity MMLU Vicuna eval GLUE ARC HellaSwag TruthfulQA Winogrande GSM8K

Metrics: perplexity accuracy win/tie/loss vs GPT-3.5 (GPT-4 judge) average GLUE score

Baselines: QLoRA GPTQ-LoRA Full FT RTN GPTQ AWQ OmniQuant SpQR SqueezeLLM

Builds on: LoRA QLoRA GPTQ NormalFloat quantization robust PCA

Closest papers by meaning

• NOLA: Compressing LoRA using Linear Combination of Random Basis (ICLR 2024 · similarity 0.91)
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models (ICLR 2024 · similarity 0.91)
• LQER: Low-Rank Quantization Error Reconstruction for LLMs (ICML 2024 · similarity 0.91)
• Dynamic Low-Rank Sparse Adaptation for Large Language Models (ICLR 2025 · similarity 0.90)
• RILQ: Rank-Insensitive LoRA-Based Quantization Error Compensation for Boosting 2-Bit Large Language Model Accuracy (AAAI 2025 · similarity 0.90)
• LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits (ICML 2025 · similarity 0.90)