Research map / Architectures and efficiency / Quantization
Scaling FP8 training to trillion-token LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains Llama2 7B in FP8 on up to 2T tokens, traces late-training divergence to SwiGLU outlier amplification via w1/w2 alignment, and proposes Smooth-SwiGLU plus FP8 Adam moments.
Problem
FP8 LLM training that is stable for short runs diverges during long (trillion-token) training, because of outliers in activations, and optimizer states are still kept in high precision.
Approach
Shows analytically that SwiGLU weight vectors w1 and w2 tend to align under l2 regularization when inputs are large, causing quadratic outlier amplification; proposes Smooth-SwiGLU, which applies per-channel scaling before quantization of the SwiGLU output and inverts it after the last linear layer, leaving the function unchanged. Also quantizes both Adam moments to FP8 (E4M3 for first moment, E5M
Key results
- Standard FP8 (SwiGLU output quantized, FP32 Adam moments) diverges after ~200B tokens, while Smooth-SwiGLU plus FP8 optimizer converges close to BF16 on 2T tokens.
- FP8 + Smooth-SwiGLU reaches +33.52% throughput over BF16 on 8 Intel Gaudi2 (16.89 vs 12.65 samples/sec), versus +37.08% for unstable full FP8.
- Zero-shot results for FP8 (2) are on par with BF16 (e.g., HellaSwag 68.37 vs 68.3; Wikitext perplexity 5.55 vs 5.59); optimizer moment quantization reduces memory from 63.25 to 44.08 GB/HPU in the Deepspeed Zero-1 setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Smooth-SwiGLU instead of standard SwiGLU (ablation) | FP8 training of Llama2 7B · Red Pajama training up to 2 trillion tokens | training stability | Helped |
| Disable quantization of the SwiGLU output (ablation) | FP8 training of Llama2 · Large-dataset training | training stability | Helped |
| Use full FP8 quantization with standard SwiGLU instead of Smooth-SwiGLU or BF16 SwiGLU output (ablation) | Llama2 7B training · Large-dataset training | training stability | Hurt |
| Use Smooth-SwiGLU instead of BF16 | Llama2 7B training on Intel Gaudi2 · Throughput | throughput: 12.65 -> 16.89 (+33.52 %) | Helped |
| Disable quantization of the SwiGLU output (ablation) | Llama2 7B training on Intel Gaudi2 · Throughput | throughput: 12.65 -> 16.07 (+ 27.04 %) | Helped |
| Replace standard SwiGLU with Smooth-SwiGLU (ablation) | FP8 training of Llama2 7B · Intel Gaudi2 | throughput: 17.34 -> 16.89 | Mixed |
| Quantize both Adam moments to FP8 | FP8 optimizer for Llama2 7B · Training memory | memory (GB/HPU): 63.25 -> 44.08 | Helped |
| Use FP8 (1), with SwiGLU output in BF16, instead of BF16 (ablation) | Llama2 7B · Lambada | accuracy: 61.98 -> 61.73 | Hurt |
| Use FP8 (2), with Smooth-SwiGLU and FP8 optimizer, instead of BF16 | Llama2 7B · Winogrande | accuracy: 64.25 -> 65.43 | Helped |
| Use FP8 (2), with Smooth-SwiGLU and FP8 optimizer, instead of BF16 | Llama2 7B · Lambada | perplexity: 5.75 -> 5.9 | Hurt |
| Use FP8 (1), with SwiGLU output in BF16, instead of BF16 (ablation) | Llama2 7B · Wikitext | perplexity: 5.59 -> 5.56 | Helped |
| Quantize the first Adam moment to E4M3 and the second moment to E5M2 (ablation) | FP8 Adam optimizer · Llama2 100m convergence | convergence | Helped |
Surprising findings
- Outlier amplification appears only after prolonged training (~200B tokens), so short FP8 runs did not reveal the instability.
- Smooth-SwiGLU also yields smoother BF16 training and lower loss at larger learning rates.
Limitations
- Throughput measured on a non-optimized implementation.
- Memory savings measured only with Deepspeed Zero-1 on 8 devices.
- Theory assumes stationary points and that sigmoid derivative is small, which is only approximately validated.
Experimental setup
Datasets: RedPajama C4
Benchmarks: Lambada HellaSwag Winogrande Arc-C Wikitext
Metrics: accuracy perplexity training loss throughput (samples/sec) TFLOPS memory (GB/HPU)
Baselines: BF16 training FP8 with SwiGLU output in BF16 standard FP8 (FP8-LM style, FP16 second moment)
Builds on: Llama 2 SwiGLU Transformer Engine delayed scaling FP8 formats (E4M3/E5M2) FP8-LM
Closest papers by meaning
- Towards Fully FP8 GEMM LLM Training at Scale (NeurIPS 2025 · similarity 0.91)
- µnit Scaling: Simple and Scalable FP8 LLM Training (ICML 2025 · similarity 0.91)
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.89)
- COAT: Compressing Optimizer states and Activations for Memory-Efficient FP8 Training (ICLR 2025 · similarity 0.87)
- Quartet: Native FP4 Training Can Be Optimal for Large Language Models (NeurIPS 2025 · similarity 0.87)
- HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.