Research map / Architectures and efficiency / Quantization
Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Quartet, a fully MXFP4 training algorithm for LLMs (all three linear-layer GEMMs in FP4) guided by a scaling-law framework that separates forward parameter efficiency and backward data efficiency, with fused Blackwell GPU kernels.
Problem
Training LLMs with all matrix multiplications in FP4 on hardware such as NVIDIA Blackwell suffers large accuracy loss, and existing 4-bit training methods either degrade accuracy or fall back to higher precision for some matmuls.
Approach
Use scaling laws to compare quantized training methods via forward parameter efficiency (tied to quantization MSE) and backward data efficiency (tied to gradient misalignment); combine QuEST-style forward quantization with a round-to-nearest backward pass using block-wise Hadamard rotations, and implement it with fused CUTLASS kernels for MXFP4 on Blackwell.
Key results
- Quartet reaches C4 validation loss 3.29 at 100x D/N on 30M Llama vs 3.66 for LUQ-INT4 (best prior), with eff_N 0.65 and eff_D 0.95.
- Quartet kernels reach up to 2x speedup over FP8 on forward pass and about 1.6x overall training speedup on RTX 5090.
- On 7B model, Quartet PTQ-free MXFP4 training gives C4 perplexity 17.77 vs 18.19 for QuaRot PTQ and 16.40 BF16.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| QuEST forward quantization instead of SR, RTN, or LSQ (ablation) | forward pass of MXFP4 training · Llama 30M C4 scaling-law fit | eff_N: 0.64 vs 0.59 (RTN) and 0.42 (SR) | Helped |
| RTN backward rounding instead of stochastic rounding or QuEST (ablation) | backward pass of MXFP4 training · Llama 30M C4 scaling-law fit | eff_D: 0.93 vs 0.88 (SR) and 0.83 (QuEST) | Helped |
| Stochastic rounding in backward pass instead of RTN (ablation) | backward pass · Llama 30M C4 validation loss over D/N | validation loss gap: RTN consistently outperforms SR, gap shrinks with longer training | Hurt |
| Full Quartet (QuEST forward + RTN backward, Hadamard transforms) vs prior 4-bit methods | end-to-end 4-bit LLM pretraining · Llama 30M C4 validation loss | validation loss: Quartet 3.29 vs LUQ-INT4 3.66 at D/N=100x | Helped |
| Jetfire and HALO adapted to FP4 training | end-to-end FP4 training · Llama 30M C4 validation loss | validation loss: Jetfire-FP4 6.76 and HALO-FP4 6.55 at D/N=100x vs Quartet 3.29 | Hurt |
| LSS INT4 training with leverage-score sampling | end-to-end 4-bit LLM pretraining · Llama 30M C4 validation loss | stability (NaN): NaN at most D/N ratios beyond 50x | Hurt |
| Quartet MXFP4 kernels vs FP8 MXFP8 CUTLASS kernels | linear layer forward pass on RTX 5090 · Llama 7B-shape layers, batch 64, seq 512 | forward speedup: up to 2x over FP8 | Helped |
| Quartet MXFP4 kernels vs FP8 for backward pass | linear layer backward pass on RTX 5090 · Llama 7B-shape layers | backward speedup: up to 1.5x over FP8 | Helped |
| Quartet MXFP4 training vs QuaRot post-training quantization | 7B model C4 perplexity with MXFP4 · C4 perplexity 7B | perplexity: 17.77 vs 18.19 for QuaRot (BF16 16.40) | Helped |
Surprising findings
- Stochastic rounding, commonly assumed best for unbiased gradients, was outperformed by round-to-nearest in the backward pass once training is long; an earlier draft's conclusion was reversed due to a bug.
- Jetfire, HALO and LSS are unstable or diverge when ported to FP4, while LUQ-INT4 is the strongest prior baseline.
Limitations
- Designed for the standard MXFP4 data type and Blackwell compute architecture; generalization to other formats is not shown.
- Experiments are limited to small models (30M-200M) with only a single 7B stability run; larger-scale distributed training is future work.
- Misalignment and efficiency metrics are approximations, since the exact unbiased rescaling of prior work is incompatible with MXFP4 group scales.
Experimental setup
Datasets: C4
Benchmarks: C4 validation loss HellaSwag WinoGrande ARC-easy RTX 5090 kernel speedups
Metrics: validation loss parameter efficiency (eff_N) data efficiency (eff_D) speedup perplexity
Baselines: LUQ-INT4 LUQ-FP4 Jetfire-FP4 HALO-FP4 LSS-INT4 FP8 (MXFP8 CUTLASS) BF16 QuaRot
Builds on: QuEST LSQ LUQ Chinchilla scaling laws CUTLASS Llama-2
Closest papers by meaning
- Optimizing Large Language Model Training Using FP4 Quantization (ICML 2025 · similarity 0.91)
- Towards Fully FP8 GEMM LLM Training at Scale (NeurIPS 2025 · similarity 0.89)
- µnit Scaling: Simple and Scalable FP8 LLM Training (ICML 2025 · similarity 0.87)
- Scaling FP8 training to trillion-token LLMs (ICLR 2025 · similarity 0.87)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.86)
- any4: Learned 4-bit Numeric Representation for LLMs (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.