Research map / Architectures and efficiency / Quantization
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces QuEST, a quantization-aware training method using Hadamard-normalized MSE-optimal Gaussian fitting and a trust gradient estimator, showing stable training down to W1A1 and 4-bit as Pareto-optimal for Llama-style LLMs up to 1.6B parameters.
Problem
Quantization-aware training of LLMs with low-bitwidth weights and activations is unstable and inaccurate; prior work placed the Pareto-optimal QAT precision at 8 bits, and gradient estimation through the straight-through estimator is biased for quantized states.
Approach
Combine a Hadamard transform with RMS normalization and MSE-optimal clipping fitted to a Gaussian for the forward quantization, and replace the STE with a trust estimator that masks gradients for entries whose quantization error exceeds a trust threshold (with a scaling fix for 1-bit).
Key results
- At 30M and 50M sizes, QuEST W1A1 reaches C4 validation loss 3.945 and 3.791, beating LSQ (3.991, 3.862), AdaBin (3.988, 3.843) and STE (5.256, 6.867).
- Fitted scaling law gives eff(4)=0.70 for QuEST INT4 and the paper reports 4-bit as Pareto-optimal in the overtraining regime, with about 50% efficiency gaps to baselines.
- QuEST INT4 1.6B model per-layer speedups over BF16 on RTX 4090 of 1.2x-2.4x, and end-to-end prefill speedups of 1.3-1.5x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use QuEST instead of the strongest listed baseline at W4A4 | Llama 30M and 50M · C4 validation | validation loss: 3.315 -> 3.272; 3.240 -> 3.135 | Helped |
| use QuEST instead of the strongest listed baseline at W3A3 | Llama 30M and 50M · C4 validation | validation loss: 3.410 -> 3.372; 3.290 -> 3.226 | Helped |
| use QuEST instead of the strongest listed baseline at W2A2 | Llama 30M and 50M · C4 validation | validation loss: 3.598 -> 3.574; 3.501 -> 3.441 | Helped |
| use QuEST instead of the strongest listed baseline at W1A1 | Llama 30M and 50M · C4 validation | validation loss: 3.988 -> 3.945; 3.843 -> 3.791 | Helped |
| remove the Hadamard transform (ablation) | QuEST INT4 · HellaSWAG | accuracy: 39.18 -> 38.03 | Hurt |
| remove the Hadamard transform (ablation) | QuEST INT4 · ARC-Easy | accuracy: 52.40 -> 52.44 | No clear effect |
| remove the Hadamard transform (ablation) | QuEST INT4 · ARC-Challenge | accuracy: 22.01 -> 22.70 | Helped |
| remove the Hadamard transform (ablation) | QuEST INT4 · PiQA | accuracy: 71.16 -> 71.11 | Hurt |
| remove the Hadamard transform (ablation) | QuEST INT4 · Winogrande | accuracy: 52.96 -> 51.38 | Hurt |
| add 2:4 sparsity (ablation) | QuEST INT4 · HellaSWAG | accuracy: 39.18 -> 36.26 | Hurt |
| add 2:4 sparsity (ablation) | QuEST INT4 · ARC-Easy | accuracy: 52.40 -> 50.46 | Hurt |
| add 2:4 sparsity (ablation) | QuEST INT4 · ARC-Challenge | accuracy: 22.01 -> 21.08 | Hurt |
Surprising findings
- Weight-only 1-bit weights are surprisingly competitive with 3-bit weights, and 2-bit weights appear Pareto-dominant in the weight-only setting.
- FP4 performs slightly worse than INT4 in the fitted scaling law, with the authors hypothesizing higher MSE for Gaussian data when clipping is allowed.
Limitations
- Evaluated only up to 1.6B parameters; scaling to much larger models is untested.
- Focused on decoder-only Llama-style architectures; encoder-decoder and other architectures not explored.
- The 1-bit case requires an ad hoc outer trust scaling factor s=1.30 tuned on small-scale experiments.
Experimental setup
Datasets: C4
Benchmarks: C4 validation loss HellaSWAG ARC-Easy ARC-Challenge PiQA Winogrande
Metrics: validation loss accuracy scaling-law parameter efficiency eff(P) per-layer and end-to-end speedup
Baselines: STE QuaRot (QAT extension) LSQ AdaBin PACT BitNet
Builds on: straight-through estimator LSQ Chinchilla scaling laws Llama 2 QuaRot
Closest papers by meaning
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.92)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.91)
- QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models (ICLR 2024 · similarity 0.91)
- SqueezeLLM: Dense-and-Sparse Quantization (ICML 2024 · similarity 0.91)
- Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.