Research map / Architectures and efficiency / Quantization
DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DuQuant, a training-light 4-bit weight-activation PTQ method for LLMs that combines greedy-searched block-wise rotation, zigzag permutation, and a second rotation to spread both massive and normal outliers.
Problem
Low-bit weight-activation post-training quantization of LLMs is degraded by normal outliers (persistent across tokens) and massive outliers (very large values in few tokens, e.g., at FFN down-projection inputs), which existing smoothing methods handle poorly.
Approach
Use prior knowledge of outlier channels to build a greedy-searched block-diagonal rotation that moves outliers to neighboring channels, then a zigzag channel permutation that balances outlier magnitudes across blocks, followed by a second rotation; the same orthogonal transforms are applied to weights so no GPTQ or gradient-based training is needed for the core method.
Key results
- Under W4A4, DuQuant (LWC variant) reaches WikiText2 PPL 6.18 on LLaMA1-7B vs 8.15 for Atom and 25.25 for SmoothQuant (FP16 5.68).
- On LLaMA1-7B W4A4 commonsense QA, DuQuant averages 61.76 vs 56.88 for Atom (FP16 64.09).
- Layer-wise pre-filling speedup of 2.08x over FP16 for LLaMA2-7B and 2.34x for LLaMA2-13B on one RTX 3090; quantizing LLaMA2-13B takes 71s on A100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DuQuant instead of Atom | LLaMA2-7B W4A4 · WikiText2 | perplexity: 8.40 -> 6.28 | Helped |
| Use DuQuant instead of Atom | LLaMA2-7B W4A4 · C4 | perplexity: 10.96 -> 7.90 | Helped |
| Use DuQuant instead of Atom | LLaMA1-7B W4A4 · Commonsense QA average | accuracy: 56.88 -> 61.76 | Helped |
| Use DuQuant instead of Atom | Vicuna-v1.5-13B W4A4 · MMLU 0-shot | accuracy: 41.07 -> 50.94 | Helped |
| Use DuQuant instead of Atom | Vicuna-v1.5-7B W4A4 · LongBench average | score: 34.34 -> 40.37 | Helped |
| Add smoothing (ablation) | Rotation 1 + permutation + Rotation 2, LLaMA2-7B W4A4 · WikiText2 | perplexity: 6.79 -> 6.28 | Helped |
| Add smoothing (ablation) | Rotation 1 + permutation + Rotation 2, LLaMA2-7B W4A4 · C4 | perplexity: 8.51 -> 7.90 | Helped |
| Add permutation and Rotation 2 (ablation) | Smooth + Rotation 1, LLaMA2-7B W4A4 · WikiText2 | perplexity: 7.92 -> 6.28 | Helped |
| Add permutation and Rotation 2 (ablation) | Smooth + Rotation 1, LLaMA2-7B W4A4 · C4 | perplexity: 10.64 -> 7.90 | Helped |
| Add Rotation 1 to smoothing (ablation) | LLaMA2-7B W4A4 · WikiText2 | perplexity: NaN -> 7.92 | Helped |
| Add Rotation 1 to smoothing (ablation) | LLaMA2-7B W4A4 · C4 | perplexity: 1379.46 -> 10.64 | Helped |
Surprising findings
- Massive outliers were found at the input of the FFN down-projection rather than only at block outputs, and smoothing alone caused NaN/very high perplexity on LLaMA2-7B (Table: smoothing only).
- Round-to-nearest RTN DuQuant matches or beats QuaRot with GPTQ on several models, suggesting GPTQ is unnecessary when outliers are managed well.
Limitations
- Phi2-2.8B shows overflow instability in query-value matmul, leaving DuQuant behind FP results there.
- Permutation and rotation require choosing block size and greedy steps; permutation applied only once for efficiency, with more permutations giving no gain.
Experimental setup
Datasets: WikiText2
Benchmarks: WikiText2 C4 PTB PIQA ARC-E ARC-C BoolQ HellaSwag WinoGrande MMLU MT-Bench LongBench
Metrics: perplexity zero-shot accuracy 5-shot accuracy win rate speedup peak memory
Baselines: SmoothQuant Outlier Suppression+ OmniQuant QLLM AffineQuant Atom QuaRot
Builds on: SmoothQuant QuaRot OmniQuant Atom
Closest papers by meaning
- Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models (ICLR 2024 · similarity 0.93)
- FlatQuant: Flatness Matters for LLM Quantization (ICML 2025 · similarity 0.91)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.91)
- OSTQuant: Refining Large Language Model Quantization with Orthogonal and Scaling Transformations for Better Distribution Fitting (ICLR 2025 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.90)
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.