Research map / Architectures and efficiency / Quantization

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

NeurIPS 2024 · Architectures and efficiency › Quantization · approach: KV cache quantization · 47 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

QuaRot applies randomized Hadamard rotations, fused into weights via computational invariance plus online Hadamards, to quantize LLaMA weights, activations, and KV cache to 4 bits end-to-end with small perplexity loss.

Problem

Activation and KV-cache quantization in LLMs is hard to do at 4 bits because of outlier features in hidden states and keys/values, and prior 4-bit methods keep outliers in higher precision or require retraining.

Approach

Apply randomized Hadamard rotations to the residual stream and attention/FFN internals; fuse them into the weight matrices using computational invariance (RMSNorm commutes with orthogonal rotations), add only 1.5 online Hadamard transforms per layer, and then quantize everything to INT4 without keeping any outlier channels in higher precision.

Key results

• 4-bit QuaRot-GPTQ on LLaMA-2-7B gives WikiText-2 perplexity 6.10 vs 5.47 FP16, and on LLaMA-2-70B 3.79 vs 3.32, with at most 0.63 perplexity loss and about 99% zero-shot accuracy retained.
• On LLaMA-2-70B, QuaRot achieves up to 3.33x prefill speedup over FP16 on an RTX 3090 with batch size 32 and sequence length 2048 (single transformer block).
• QuaRot with round-to-nearest yields lossless 6- and 8-bit quantization of LLaMA-2 models (e.g., 70B INT8 perplexity 3.33 vs 3.32 FP16).

Measured effects reported in the paper

Surprising findings

• Hadamard rotation improves weight-only quantization too: QuaRot-GPTQ at 4-bit weights improves perplexity by up to 2.65 over GPTQ, and it also helps 2-bit weight-only quantization.
• Keys are more sensitive to KV cache quantization than values; 4-bit keys with 3-bit values beats 3-bit keys with 4-bit values.

Limitations

• Perplexity gap to FP16 remains noticeable for smaller models (e.g., 0.63 on LLaMA-2-7B) and larger for LLaMA-3 (8B perplexity 8.16 vs 6.14 with GPTQ), indicating sensitivity.
• Residual stream is not quantized; the authors list quantizing residuals and mixture-of-experts support as open work.
• Kernel speedups are measured on a single transformer block on RTX 3090 rather than end-to-end models, and decoding speedups are not always realized for small batch sizes (4-bit KV cache is slower at small batches).

Experimental setup

Datasets: WikiText-2

Benchmarks: WikiText-2 PIQA WinoGrande HellaSwag ARC-Easy ARC-Challenge LAMBADA

Metrics: perplexity zero-shot accuracy prefill speedup peak memory saving

Baselines: SmoothQuant OmniQuant QUIK Atom LLM.int8() KVQuant KIVI

Builds on: computational invariance (SliceGPT) QuIP QuIP# GPTQ

Closest papers by meaning

• SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.90)
• ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.89)
• KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (NeurIPS 2024 · similarity 0.88)
• Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.88)
• CodeGEMM: A Codebook-Centric Approach to Efficient GEMM in Quantized LLMs (NeurIPS 2025 · similarity 0.88)
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (NeurIPS 2024 · similarity 0.88)