Research map / Architectures and efficiency / Quantization
FlatQuant: Flatness Matters for LLM Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlatQuant, a post-training W4A4 quantization method that learns per-layer Kronecker-decomposed affine transformations (with scaling and learnable clipping) to flatten weight and activation distributions, plus a fused kernel for speedup.
Problem
Low-bit (W4A4) post-training quantization of LLMs suffers from outliers in weights and activations; existing pre-quantization transformations (per-channel scaling, Hadamard rotations) do not produce sufficiently flat distributions and add inference overhead.
Approach
Learn a distinct invertible affine transformation per linear layer, parameterized as a Kronecker product of two small matrices (with SVD-based inversion), plus per-channel scaling and learnable clipping, calibrated block-wise with an MSE objective, and fuse the transformation with quantization into one Triton kernel.
Key results
- On LLaMA-3-70B with W4A4 RTN, FlatQuant reports 0.94% zero-shot accuracy loss (avg 79.01 vs 79.95 FP16), surpassing SpinQuant (65.66).
- On LLaMA-3-8B W4A4 RTN, WikiText-2 PPL is 6.98 vs 6.14 FP16 and 7.96 for SpinQuant RTN.
- With kernel fusion on RTX 3090, FlatQuant achieves up to 2.30x prefill and 1.76x decoding speedup over FP16 at batch size 64.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add learnable transformation (ablation) | FlatQuant on LLaMA-3-8B · WikiText-2 | perplexity: 1266.60 -> 8.50 | Helped |
| Add per-channel scaling to learnable transformation (ablation) | FlatQuant on LLaMA-3-8B · WikiText-2 | perplexity: 8.50 -> 7.95 | Helped |
| Add learnable clipping thresholds to learnable transformation (ablation) | FlatQuant on LLaMA-3-8B · WikiText-2 | perplexity: 8.50 -> 7.11 | Helped |
| Add per-channel scaling to learnable transformation (ablation) | FlatQuant on LLaMA-3-8B · zero-shot QA | average accuracy: 66.82 -> 67.08 | Helped |
| Add learnable clipping thresholds to learnable transformation (ablation) | FlatQuant on LLaMA-3-8B · zero-shot QA | average accuracy: 66.82 -> 70.72 | Helped |
| Use SVD with AMP for affine matrix inversion instead of FP32 training with direct inverse (ablation) | FlatQuant on LLaMA-3-8B · calibration | training time: 2.2 hours -> 0.9 hours | Helped |
| Use FlatQuant instead of SpinQuant with RTN | LLaMA-3-8B W4A4 · WikiText-2 | perplexity: 7.96 -> 6.98 | Helped |
| Use FlatQuant instead of SpinQuant with RTN | LLaMA-3-8B W4A4 · C4 | perplexity: 13.45 -> 11.13 | Helped |
| Fuse affine transformation and quantization into a single kernel | FlatQuant on LLaMA-2-7B · batch size 64 prefill | speedup: +2.30x speedup | Helped |
| Fuse affine transformation and quantization into a single kernel | FlatQuant on LLaMA-2-7B · batch size 64 decoding | speedup: +1.76x speedup | Helped |
Surprising findings
- Learnable affine transformations with RTN weight quantization perform comparably to those combined with GPTQ, reducing calibration cost.
Limitations
- Calibration still needs gradient-based block-wise training (hours per model, e.g. 5.94 hours for LLaMA-3-70B), and the speedup gap to vanilla INT4 remains.
- Speedup claims are measured on an RTX 3090 only.
- Kronecker size selection affects speed; very large sizes cause slowdowns from irregular memory access.
Experimental setup
Datasets: WikiText-2 C4
Benchmarks: WikiText-2 perplexity C4 perplexity ARC-Challenge ARC-Easy HellaSwag LAMBADA PIQA Winogrande C-Eval MMLU AIME2024 MT-Bench
Metrics: perplexity zero-shot accuracy average accuracy prefill latency speedup decoding speedup MSE
Baselines: SmoothQuant OmniQuant AffineQuant QUIK-4B QuaRot SpinQuant AWQ QuIP
Builds on: QuaRot SpinQuant SmoothQuant AffineQuant GPTQ
Closest papers by meaning
- AffineQuant: Affine Transformation Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.91)
- DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs (NeurIPS 2024 · similarity 0.91)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.90)
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.90)
- OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.