AI Paper Atlas

Research map / Architectures and efficiency / Quantization

Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

Andrei Panferov, Erik Schultheis, Soroush Tabesh, Dan Alistarh

ICML 2026 · Architectures and efficiency › Quantization · approach: Ultra-low-bit LLM quantization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace stochastic rounding with MS-EDEN for unbiased backward quantization (ablation)Quartet II · Quadratic error on N(0,1)MSE x1e-3: 23.5 -> 9.4Improved
replace SR+4/6 with MS-EDEN (ablation)NVFP4 backward · Quadratic error on N(0,1)MSE x1e-3: 17.5 -> 9.4Improved
MS-EDEN instead of SR on backward GEMMs (ablation)NVFP4 pre-training · C4 val loss, Llama-2-like 30M-100Mvalidation loss increase vs BF16Improved
MS-EDEN with weight re-quantization instead of SR without it (ablation)NVFP4 pre-training · C4 val loss, Llama-2-likevalidation loss increase vs BF16Improved
add 4/6 scale selection to native 1x16 scales on forward pass (ablation)NVFP4 forward · C4 val loss, Llama-2-like 30M-200Mvalidation loss gap vs BF16: 7.6 vs 9.0 MSE x1e-3Improved
add 4/6 to square-block 16x16 scales (ablation)NVFP4 forward · Quadratic error on N(0,1)MSE x1e-3: 12.4 -> 12.4No clear effect
use native 1x16 scales instead of square 16x16 blocks on forward pass (ablation)NVFP4 forward · Quadratic error on N(0,1)MSE x1e-3: 12.4 -> 9.0Improved
use Quartet II instead of NVIDIA, 4/6, TetraJet-v2 recipesNVFP4 full quantization · C4 val loss, Llama-2-likevalidation loss gap vs BF16: at least 20% lower loss gapImproved
use Quartet II instead of existing NVFP4 methodsNVFP4 Nanochat pre-training · FineWeb-Edu validation BPBBPB gap vs BF16: 15-25% smaller gapImproved
use Quartet II instead of other QAT methods after SFTNVFP4 Nanochat · ARC, GSM8K zero-shotaccuracyNo clear effect
post hoc range alignment instead of naive range alignment (ablation)MS-EDEN kernel · GMEM bandwidthbandwidth: around 20% theoretical savingImproved
apply 4/6 with SR on backward passFourOverSix · gradient estimationbiasHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy