Research map / Architectures and efficiency / Quantization
Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation
Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace stochastic rounding with MS-EDEN for unbiased backward quantization (ablation) | Quartet II · Quadratic error on N(0,1) | MSE x1e-3: 23.5 -> 9.4 | Improved |
| replace SR+4/6 with MS-EDEN (ablation) | NVFP4 backward · Quadratic error on N(0,1) | MSE x1e-3: 17.5 -> 9.4 | Improved |
| MS-EDEN instead of SR on backward GEMMs (ablation) | NVFP4 pre-training · C4 val loss, Llama-2-like 30M-100M | validation loss increase vs BF16 | Improved |
| MS-EDEN with weight re-quantization instead of SR without it (ablation) | NVFP4 pre-training · C4 val loss, Llama-2-like | validation loss increase vs BF16 | Improved |
| add 4/6 scale selection to native 1x16 scales on forward pass (ablation) | NVFP4 forward · C4 val loss, Llama-2-like 30M-200M | validation loss gap vs BF16: 7.6 vs 9.0 MSE x1e-3 | Improved |
| add 4/6 to square-block 16x16 scales (ablation) | NVFP4 forward · Quadratic error on N(0,1) | MSE x1e-3: 12.4 -> 12.4 | No clear effect |
| use native 1x16 scales instead of square 16x16 blocks on forward pass (ablation) | NVFP4 forward · Quadratic error on N(0,1) | MSE x1e-3: 12.4 -> 9.0 | Improved |
| use Quartet II instead of NVIDIA, 4/6, TetraJet-v2 recipes | NVFP4 full quantization · C4 val loss, Llama-2-like | validation loss gap vs BF16: at least 20% lower loss gap | Improved |
| use Quartet II instead of existing NVFP4 methods | NVFP4 Nanochat pre-training · FineWeb-Edu validation BPB | BPB gap vs BF16: 15-25% smaller gap | Improved |
| use Quartet II instead of other QAT methods after SFT | NVFP4 Nanochat · ARC, GSM8K zero-shot | accuracy | No clear effect |
| post hoc range alignment instead of naive range alignment (ablation) | MS-EDEN kernel · GMEM bandwidth | bandwidth: around 20% theoretical saving | Improved |
| apply 4/6 with SR on backward pass | FourOverSix · gradient estimation | bias | Hurt |
Closest papers by meaning
- Quartet: Native FP4 Training Can Be Optimal for Large Language Models (NeurIPS 2025 · similarity 0.90)
- RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.88)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.87)
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.87)
- LoQT: Low-Rank Adapters for Quantized Pretraining (NeurIPS 2024 · similarity 0.87)
- GANQ: GPU-Adaptive Non-Uniform Quantization for Large Language Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
