Research map / Architectures and efficiency / Quantization
Optimizing Large Language Model Training Using FP4 Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the first FP4 (E2M1) pretraining framework for LLMs, combining a differentiable gradient estimator for weights and outlier clamping-and-compensation for activations, simulated on H100 FP8 tensor cores and evaluated on LLaMA models up to 13B trained on 100B tokens.
Problem
Training LLMs in 4-bit floating point is hard because of large quantization errors, a limited dynamic range, and activation outliers; no prior FP4 pretraining framework for LLMs existed.
Approach
Replace the straight-through estimator in the backward pass with a differentiable gradient estimator (DGE) derived from a smooth power-based approximation of the E2M1 quantizer, and handle activation outliers by quantile clamping plus a sparse high-precision compensation matrix, combined with vector-wise scaling and mixed-precision training.
Key results
- FP4 training reaches 2.55 vs 2.49 BF16 training loss for 1.3B and 1.97 vs 1.88 for 13B after 100B tokens.
- Zero-shot average accuracy of FP4 models is comparable to BF16: 53.13 vs 53.23 (1.3B), 54.42 vs 53.87 (7B), 54.95 vs 54.44 (13B).
- Directly casting activations to FP4 (W8A4 without OCC) causes loss divergence to NaN in ablations, while OCC with quantile 0.99 keeps convergence.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with FP4 instead of BF16 | LLaMA 2 1.3B · DCLM, 100B tokens | training loss: 2.49 -> 2.55 | Hurt |
| Train with FP4 instead of BF16 | LLaMA 2 7B · DCLM, 100B tokens | training loss: 2.07 -> 2.17 | Hurt |
| Train with FP4 instead of BF16 | LLaMA 2 13B · DCLM, 100B tokens | training loss: 1.88 -> 1.97 | Hurt |
| Train with FP4 instead of BF16 | LLaMA 2 models · Zero-shot average, 1.3B / 7B / 13B | accuracy: 53.23 -> 53.13; 53.87 -> 54.42; 54.44 -> 54.95 | Mixed |
| Train with FP4 instead of BF16 | LLaMA 2 models · Perplexity average, 1.3B / 7B / 13B | perplexity: 37.38 -> 36.86; 35.06 -> 35.62; 33.69 -> 33.99 | Mixed |
| Add activation clamping (ablation) | FP4 activation quantization · LLaMA 1.3B activation tensors, 30,000 iterations | cosine similarity: 92.19% -> 98.83% | Helped |
| Add sparse outlier compensation after clamping (ablation) | FP4 activation quantization · LLaMA 1.3B activation tensors, quantile 99.9 | cosine similarity: 98.83% -> 99.61% | Helped |
| Lower compensation quantile from 99.9 to 99 (ablation) | FP4 activation quantization · LLaMA 1.3B activation tensors | cosine similarity: 99.61% -> 100% | Helped |
| Lower compensation quantile from 99.9 to 97 (ablation) | FP4 activation quantization · LLaMA 1.3B activation tensors | cosine similarity: 99.61% -> 100% | Helped |
| Use DGE for weight-only 4-bit quantization instead of direct quantization (ablation) | W4A8 training · LLaMA 1.3B, DCLM subset, 10B tokens | training loss | Helped |
| Use direct FP4 activation quantization without OCC (ablation) | W8A4 training · LLaMA 1.3B, DCLM subset, 10B tokens | training loss | Hurt |
| Add OCC to FP4 activation quantization (ablation) | W8A4 training · LLaMA 1.3B, DCLM subset, 10B tokens | training loss | Helped |
Surprising findings
- FP4 models score slightly higher than BF16 on some zero-shot averages at 7B and 13B despite a higher training loss.
- Weight-only 4-bit quantization without DGE shows no substantial training loss gap, suggesting weights are easier to quantize than activations.
Limitations
- No native FP4 tensor cores, so speedups are not measured and FP4 is simulated on FP8 hardware with added overhead.
- Experiments do not extend to very large models or trillion-token datasets.
- Sparse outlier compensation is inefficient on current hardware, requiring a high clamp quantile.
Experimental setup
Datasets: DCLM
Benchmarks: Arc-C Arc-E BoolQ HellaSwag LogiQA PiQA SciQ OpenbookQA Lambada Pile 10k Wikitext
Metrics: training loss zero-shot accuracy perplexity cosine similarity MSE SNR
Baselines: BF16 mixed-precision training MS-AMP FP8 Transformer Engine FP8 directly-casted W4A4 FP4
Builds on: Straight-Through Estimator MS-AMP Transformer Engine FP8 LLaMA 2
Closest papers by meaning
- Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs (ICML 2024 · similarity 0.91)
- Quartet: Native FP4 Training Can Be Optimal for Large Language Models (NeurIPS 2025 · similarity 0.91)
- Norm Tweaking: High-Performance Low-Bit Quantization of Large Language Models (AAAI 2024 · similarity 0.91)
- BiLLM: Pushing the Limit of Post-Training Quantization for LLMs (ICML 2024 · similarity 0.90)
- What Makes Quantization for Large Language Model Hard? An Empirical Study from the Lens of Perturbation (AAAI 2024 · similarity 0.90)
- QuEST: Stable Training of LLMs with 1-Bit Weights and Activations (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.