Research map / Architectures and efficiency / Quantization
SpinQuant: LLM Quantization with Learned Rotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns orthogonal rotation matrices (via Cayley SGD on the Stiefel manifold) that are absorbed into LLM weights to remove outliers, substantially improving 4-bit weight/activation/KV-cache post-training quantization.
Problem
Post-training quantization of LLM weights, activations, and KV cache to low bit-widths suffers large errors from outliers; random rotations help but their quantization outcome varies widely (up to 13 points) across random draws.
Approach
Insert rotation matrices R1/R2 that are rotationally invariant in full precision (absorbed into weights, plus online Hadamard R3/R4 for extreme quantization), and optimize them with Cayley SGD on the Stiefel manifold to minimize the quantized network's task loss, rather than using random rotations.
Key results
- With 4-bit weight, activation, and KV-cache quantization, SpinQuant (SpinQuant_had) reaches 64.0 average zero-shot accuracy on LLaMA-2 7B, a 2.9-point gap to full precision (66.9), outperforming LLM-QAT by 19.1 points.
- On LLaMA-3 8B at 4-4-4 vs QuaRot+GPTQ, SpinQuant reduces the gap to full precision substantially (65.5 vs 65.3 for QuaRot... reported as up to 45.1% relative gap reduction).
- Learned rotations beat random Hadamard rotations by up to 16.2 points on Mistral-7B at 4-4-4 (68.6 vs 52.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace random Hadamard R1,R2 with learned R1,R2 (ablation) | SpinQuant on LLaMA-3.2 3B · W4A4KV4 zero-shot commonsense average | accuracy: 49.6 -> 52.9 | Helped |
| Replace random Hadamard R1,R2,R3,R4 with learned rotations (ablation) | SpinQuant on Mistral-7B · W4A4KV4 zero-shot commonsense average | accuracy: 52.4 -> 68.6 | Helped |
| Add online Hadamard rotations R3,R4 to learned R1,R2 (ablation) | SpinQuant on LLaMA-2 7B · W4A4KV4 zero-shot commonsense average | accuracy: 56.0 -> 64.0 | Helped |
| Optimize rotations using activations-only quantization rather than weights-and-activations quantization (ablation) | SpinQuant on LLaMA-2 7B · W4A4KV4 zero-shot commonsense average | accuracy: 60.9 $_{\pm 0.6}$ -> 64.0 $_{\pm 0.3}$ | Helped |
| Optimize rotations with Cayley SGD rather than leave them random (ablation) | SpinQuant on LLaMA-2 7B · W4A4 zero-shot commonsense average | accuracy: 58.7 $_{\pm 1.0}$ -> 61.5 $_{\pm 0.3}$ | Helped |
| Initialize Cayley optimization with floating-point instead of Hadamard rotations (ablation) | SpinQuant on LLaMA-2 7B · W4A4 zero-shot commonsense average | accuracy: 61.5 $_{\pm 0.3}$ vs 61.5 $_{\pm 0.8}$ | No clear effect |
| Add online Hadamard rotations (ablation) | SpinQuant on LLaMA-3 8B · W4A8 decoding | latency: 58.88 ms/token -> 63.90 ms/token | Hurt |
| Use optimized rotations instead of QuaRot rotations | SpinQuant + GPTQ on LLaMA-3 8B · W4A4KV4 zero-shot commonsense average | accuracy: 63.3 -> 65.5 | Helped |
Surprising findings
- Random rotations give a 13-point spread in zero-shot accuracy for W4A4 LLaMA-2 7B across 100 random seeds, so the choice of random rotation matters greatly.
- Optimizing rotations with respect to an activation-only quantized network (weights kept 16-bit, GPTQ applied afterward) worked better than optimizing with weights quantized too.
Limitations
- Online Hadamard rotations R3/R4 require kernel support and add some latency overhead.
- Rotation optimization is applied per model and uses a calibration set, so it adds a preprocessing cost (up to 3.5 hours for 70B).
Experimental setup
Datasets: WikiText-2
Benchmarks: BoolQ PIQA SIQA HellaSwag WinoGrande ARC-easy ARC-challenge OpenBookQA WikiText-2 perplexity
Metrics: zero-shot accuracy average perplexity
Baselines: RTN SmoothQuant LLM-QAT GPTQ OmniQuant AWQ QuIP# QuaRot
Builds on: QuaRot SliceGPT Cayley SGD GPTQ LLM-QAT
Closest papers by meaning
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
- DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization (NeurIPS 2025 · similarity 0.90)
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (NeurIPS 2024 · similarity 0.90)
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (NeurIPS 2024 · similarity 0.90)
- ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs (NeurIPS 2026 · similarity 0.90)
- FlatQuant: Flatness Matters for LLM Quantization (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.