Research map / Architectures and efficiency / Quantization
ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ResQ, a PTQ method that keeps a PCA-identified 1/8-dimensional high-variance subspace of activations, weights and KV cache in 8-bit and quantizes the rest to 4-bit after random orthogonal rotations, without gradient-based training.
Problem
Post-training quantization of LLMs to 4-bit weights, activations and KV cache degrades accuracy badly because of extreme activation outliers, and existing outlier-keeping or rotation-based methods leave a notable gap to 16-bit performance.
Approach
Combine PCA-based mixed precision (keep the highest-variance low-rank subspace in 8-bit) with random orthogonal rotations applied within each precision group, and prove that the PCA choice minimizes expected quantization error; projections are fused into adjacent weights so runtime overhead is small.
Key results
- On Meta-Llama-3-8B at W4/A4/KV4, ResQ reaches 7.1 Wikitext perplexity versus 7.4 for SpinQuant and 6.1 for the 16-bit baseline.
- ResQ achieves up to 3.03x speedup over the 16-bit baseline for a single decoder block on an RTX 3090.
- On Qwen2.5-7B at W4/A4/KV4, ResQ gets 8.2 Wikitext perplexity while QuaRot reaches 4035.9 and QUIK 260.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ResQ instead of SpinQuant | Meta-Llama-3-8B · Wikitext, W/A/KV 4/4/4 | perplexity: 7.4 -> 7.1 | Helped |
| Use ResQ instead of SpinQuant | Meta-Llama-3-8B · Average 0-shot common sense, W/A/KV 4/4/4 | accuracy: 63.8 -> 63.9 | Helped |
| Use ResQ instead of SpinQuant | Meta-Llama-3-8B · MMLU, W/A/KV 4/4/4 | accuracy: 56.2 -> 57.2 | Helped |
| Use ResQ instead of QuaRot | Meta-Llama-3-70B · Wikitext, W/A/KV 4/4/4 | perplexity: 5.7 -> 4.1 | Helped |
| Use ResQ instead of SpinQuant | Llama-3.2-3B · Wikitext, W/A/KV 4/4/4 | perplexity: 9.2 -> 8.8 | Helped |
| Use ResQ instead of SpinQuant | Meta-Llama-3-8B · GSM8K 5-shot strict match | accuracy: 29.6 -> 33.2 | Helped |
| Use ResQ instead of QUIK | Qwen2-VL-7B-Instruct · MMMU, W/A/KV 4/4/4 | accuracy: 28.9 -> 47.0 | Helped |
| Add the U_D projection (ablation) | ResQ on Llama-2-7b-hf · Wikitext, W/A/KV 4/4/4 | perplexity: 1550 -> 5.8 | Helped |
| Add the U_A projection (ablation) | ResQ on Llama-2-7b-hf · Wikitext, W/A/KV 4/4/4 | perplexity: 2500 -> 5.8 | Helped |
| Add the U_B projection (ablation) | ResQ on Llama-2-7b-hf · Wikitext, W/A/KV 4/4/4 | perplexity: 5.8 -> 5.8 | No clear effect |
| Add the U_C projection (ablation) | ResQ on Meta-Llama-3-8B · Wikitext, W/A/KV 4/4/4 | perplexity: 7.3 -> 7.1 | Helped |
| Add both U_B and U_C projections (ablation) | ResQ on Llama-3.2-3B · Wikitext, W/A/KV 4/4/4 | perplexity: 9.4 -> 8.8 | Helped |
Surprising findings
- Removing the U_A projection increases Llama-2-7b perplexity from 5.8 to 2500, showing that some projections are critical.
Limitations
- Gap to 16-bit remains on some tasks, e.g. GSM8K and MMLU for smaller models.
- Relies on GPTQ for weight quantization and on Hadamard matrices for the down_proj rotation, with random orthogonal fallback where Hadamard is unavailable.
Experimental setup
Datasets: WikiText
Benchmarks: WikiText perplexity ARC-c ARC-e BoolQ HellaSwag OpenBookQA PIQA SIQA WinoGrande MMLU GSM8K LongBench
Metrics: perplexity accuracy speedup
Baselines: RTN GPTQ SmoothQuant+ QUIK QuaRot SpinQuant
Builds on: QuaRot SpinQuant QUIK GPTQ SliceGPT
Closest papers by meaning
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.92)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.92)
- CBQ: Cross-Block Quantization for Large Language Models (ICLR 2025 · similarity 0.92)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.91)
- Evaluating Quantized Large Language Models (ICML 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.