Research map / Architectures and efficiency / Quantization
DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DartQuant, a distribution-aware rotation calibration for LLM quantization that uses a Whip loss to push rotated activations toward uniform and QR-Orth to optimize orthogonal matrices cheaply, cutting 70B rotation calibration cost sharply.
Problem
End-to-end fine-tuning of rotation matrices for LLM post-training quantization is computationally expensive, memory-hungry, requires manifold-aware orthogonal optimizers, and overfits small calibration sets.
Approach
Reformulate rotation optimization as transforming rotated activations toward a uniform (outlier-compressed) distribution via a Whip loss sum(exp(-|x_i|)), and enforce orthogonality by QR-decomposing an unconstrained latent matrix so any standard optimizer (SGD/Adam) can be used.
Key results
- On Llama-3 70B at W4A4KV16, DartQuant averages 69.39 zero-shot accuracy versus 67.94 for OSTQuant and 66.06 for SpinQuant (the paper reports about 3.31% average loss versus FP).
- For the 70B model, rotation calibration takes 0.91 GPU hours versus 42.9 (SpinQuant) and 44.0 (OSTQuant), and 23.47 GiB memory versus 238.89 and 583.86 GiB, reported as 47x speedup and 10x memory savings.
- DartQuant completes 70B rotation calibration on a single RTX 3090 in about 2.9 GPU hours with the same 23.47 GiB memory footprint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| calibrate rotations with Whip loss instead of quantization loss (ablation) | DartQuant · activation distribution calibration | activation quantization loss | Helped |
| use variance as the rotation optimization objective (ablation) | DartQuant · activation distribution calibration | activation quantization loss | No clear effect |
| use kurtosis as the rotation optimization objective (ablation) | DartQuant · activation distribution calibration | activation quantization loss | No clear effect |
| use QR-Orth instead of Cayley optimization with SGD (ablation) | rotation optimization · 100 optimization iterations | time: 8.2h -> 5.7h | Helped |
| use QR-Orth instead of Cayley optimization with Adam (ablation) | rotation optimization · 100 optimization iterations | time: 8.1h -> 5.7h | Helped |
| use QR-Orth with SGD rather than Cayley SGD (ablation) | rotation optimization · same result as Cayley SGD after 100 steps | steps to reach the result: 100 steps -> 6 steps | Helped |
| change the calibration dataset among WikiText2, PTB, and C4 (ablation) | DartQuant on Llama-2 7B · calibration dataset comparison | average PPL: 18.85; 18.90; 18.97 | No clear effect |
| change the calibration dataset among WikiText2, PTB, and C4 (ablation) | DartQuant on Llama-2 13B · calibration dataset comparison | average PPL: 23.61; 23.68; 23.59 | No clear effect |
Surprising findings
- SpinQuant and OSTQuant achieve lower perplexity than their zero-shot accuracy would suggest, which the authors attribute to overfitting of end-to-end fine-tuning on calibration data.
- Training rotations with quantization loss, variance, or kurtosis barely changes the activation distribution compared with random Hadamard rotation, whereas the Whip loss does.
Limitations
- Gains over SpinQuant and OSTQuant on perplexity are small or absent in several settings (e.g., some W4A4 perplexity entries are worse than OSTQuant), so the accuracy advantage is mostly on zero-shot tasks.
- Evaluation focuses on Llama-family models and two MoE models; the method's generalization to other architectures is not extensively shown.
Experimental setup
Datasets: WikiText2 C4 PTB
Benchmarks: WikiText2 C4 PTB LAMBADA HellaSwag PIQA WinoGrande OpenBookQA SIQA MMLU ARC-E ARC-C
Metrics: perplexity zero-shot accuracy GPU hours memory (GiB)
Baselines: RTN SmoothQuant GPTQ OmniQuant QuaRot SpinQuant OSTQuant
Builds on: SpinQuant QuaRot OSTQuant GPTQ Cayley SGD
Closest papers by meaning
- SpinQuant: LLM Quantization with Learned Rotations (ICLR 2025 · similarity 0.90)
- GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling (NeurIPS 2026 · similarity 0.89)
- LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Grid (ICLR 2025 · similarity 0.89)
- ConQuR: Corner Aligned Activation Quantization via Optimized Rotations for LLMs (NeurIPS 2026 · similarity 0.88)
- ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models (AAAI 2025 · similarity 0.88)
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.