Research map / Architectures and efficiency / Quantization

Compressing Large Language Models using Low Rank and Low Precision Decomposition

NeurIPS 2024 · Architectures and efficiency › Quantization · approach: KV cache quantization · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CALDERA, a calibration-aware post-training compression that decomposes each LLM weight matrix as a low-precision backbone plus low-precision low-rank factors (W ≈ Q + LR), with optional LoRA fine-tuning of the factors.

Problem

Deploying large LLMs on memory-constrained edge devices requires compression; existing post-training quantization at sub-2.5 bits per parameter degrades zero-shot performance, and low-rank factors are usually kept in high precision.

Approach

Jointly optimize a quantized backbone Q and low-precision low-rank factors L, R (calibration-aware, minimizing ||(Q+LR-W)X^T||_F^2) by alternating LDLQ quantization of the residual with a closed-form, quantization-aware rank-constrained regression for L and R; build on QuIP# and LoftQ-style LoRA fine-tuning.

Key results

• On LLaMa-2 70B at ~2.1 avg bits, CALDERA (rank 128, 4-bit factors) reaches Wikitext2 perplexity 4.11 vs 4.16 for QuIP# at 2.4 bits (no FT).
• On LLaMa-3 8B, rank-256 CALDERA with 4-bit factors (2.4 bits) reduces Wikitext2 perplexity to 8.22 vs 13.8 for QuIP# at 2 bits.
• LoRA fine-tuning of rank-64 factors on LLaMa-2 7B at 2.4 bits gives Wikitext2 perplexity 6.06 vs 7.85 for LoftQ at 2.4 bits, with fewer bits.

Measured effects reported in the paper

Surprising findings

• Zero-shot accuracy is noisy and not directly indicative of quantization quality, so perplexity is emphasized.
• LLaMa-3 8B degrades more under 2-bit quantization than LLaMa-2 series, with CALDERA gains smaller relative to unquantized model.

Limitations

• Throughput is reduced relative to QuIP# due to dequantizing and multiplying low-rank factors, with no custom kernels.
• Zero-shot accuracy results are noisy and sometimes not monotone with perplexity.
• Fine-tuned factors are stored in 16-bit, adding memory overhead.

Experimental setup

Datasets: RedPajama WikiText2 C4 RTE WinoGrande

Benchmarks: WikiText2 C4 Winogrande RTE PIQA ARC-Easy ARC-Challenge

Metrics: perplexity zero-shot accuracy throughput (tokens/s)

Baselines: QuIP# LoftQ LQ-LoRA unquantized models

Builds on: QuIP# LDLQ LPLR LoftQ LoRA

Closest papers by meaning

• 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs (NeurIPS 2025 · similarity 0.93)
• SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression (AAAI 2025 · similarity 0.92)
• MoDeGPT: Modular Decomposition for Large Language Model Compression (ICLR 2025 · similarity 0.91)
• LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.91)
• LowRankArena: A Standardized Evaluation Platform for SVD-Based LLM Compression (NeurIPS 2026 · similarity 0.90)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.90)