Research map / Architectures and efficiency / Quantization
Oscillation-Reduced MXFP4 Training for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TetraJet, a fully MXFP4 (forward and backward) transformer training method with truncation-free scaling, double quantization and stochastic rounding for gradients, plus Q-EMA and Q-Ramping to reduce weight oscillation in the forward pass, evaluated on DeiT and Swin pre-training on ImageNet-1K.
Problem
Pre-training transformers with MXFP4 (4-bit microscaling) causes significant accuracy degradation, and the source of this degradation has not been systematically identified.
Approach
Identifies weight oscillation of latent weights around quantization thresholds in the forward pass as the main source of MXFP4 training degradation, and proposes an EMA-based quantizer (Q-EMA) that uses a slowly-moving EMA weight to choose rounding, and an adaptive optimizer (Q-Ramping) that enlarges batch size and learning rate for oscillating weights; the TetraJet linear layer makes all six matm
Key results
- TetraJet+Q-EMA reaches 77.32% top-1 on DeiT-B vs 74.54% for Microscaling baseline and 75.57% full precision.
- Q-EMA and Q-Ramping reduce the rate of change of quantized weights (0.0045 to 0.0018 and 0.0028) on DeiT-T, and the paper claims more than 50% reduction in accuracy degradation vs the baseline.
- On DeiT-T, TetraJet reaches 59.75% top-1 vs 58.56% for Microscaling and 63.73% full precision; Q-Ramping reaches 60.31%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TetraJet instead of Microscaling | MXFP4 Vision Transformer pre-training · ImageNet-1K validation, DeiT-T/DeiT-S/DeiT-B/Swin-T/Swin-S | Top-1 accuracy: 58.56 -> 59.75; 70.10 -> 71.03; 74.54 -> 74.91; 76.87 -> 77.12; 79.45 -> 79.51 | Helped |
| Add Q-EMA | TetraJet · ImageNet-1K validation, DeiT-T/DeiT-S/DeiT-B/Swin-T/Swin-S | Top-1 accuracy: 59.75 -> 60.00; 71.03 -> 72.25; 74.91 -> 77.32; 77.12 -> 77.30; 79.51 -> 79.74 | Helped |
| Add Q-Ramping | TetraJet · ImageNet-1K validation, DeiT-T/DeiT-S/DeiT-B/Swin-T/Swin-S | Top-1 accuracy: 59.75 -> 60.31; 71.03 -> 71.32; 74.91 -> 75.62; 77.12 -> 77.33; 79.51 -> 79.67 | Helped |
| Add Dampen (ablation) | TetraJet · ImageNet-1K validation, DeiT-T and DeiT-S | Top-1 accuracy: 59.75 -> 59.75; 71.03 -> 70.75 | Mixed |
| Add Freeze (ablation) | TetraJet · ImageNet-1K validation, DeiT-T and DeiT-S | Top-1 accuracy: 59.75 -> 16.45; 71.03 -> 22.04 | Hurt |
| Remove forward weight quantization (ablation) | TetraJet · ImageNet-1K validation, DeiT-B | Top-1 accuracy: 74.91 -> 75.16 | Helped |
| Remove forward weight and activation quantization (ablation) | TetraJet · ImageNet-1K validation, DeiT-B | Top-1 accuracy: 74.91 -> 75.86 | Helped |
| Replace truncation-free scaling with Microscaling scaling while retaining stochastic rounding and double quantization (ablation) | TetraJet · ImageNet-1K validation, DeiT-T | Top-1 accuracy: 59.75 -> 59.18 | Hurt |
| Replace double quantization with Microscaling's gradient computation while retaining stochastic rounding and truncation-free scaling (ablation) | TetraJet · ImageNet-1K validation, DeiT-T | Top-1 accuracy: 59.75 -> 56.98 | Hurt |
| Use deterministic instead of stochastic backward rounding (ablation) | TetraJet with double quantization and truncation-free scaling · ImageNet-1K validation, DeiT-T | Top-1 accuracy: 59.75 -> 58.60 | Hurt |
| Use E3M0 rather than E2M1 for weights and activations, retaining E2M1 gradients (ablation) | TetraJet · ImageNet-1K validation, DeiT-T | Top-1 accuracy: 59.75 -> 54.21 | Hurt |
| Use E3M0 rather than E2M1 for gradients, retaining E2M1 weights and activations (ablation) | TetraJet · ImageNet-1K validation, DeiT-T | Top-1 accuracy: 59.75 -> 58.90 | Hurt |
Surprising findings
- Freeze oscillation method collapses in MXFP4 pre-training (16.45% on DeiT-T), and Dampen gives no gain.
- Forward-pass weight and activation quantizers cause most of the degradation, while individual backward quantizers have little effect.
Limitations
- Experiments only on image classification ViTs (DeiT, Swin) on ImageNet-1K, not LLMs or other modalities.
- Q-EMA and Q-Ramping cannot be trivially combined; the paper reports combined results only through separate runs.
- Q-Ramping requires periodic oscillation detection and hyperparameter choices (k1, k2).
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy top-5 accuracy rate of change oscillation ratio quantization confidence
Baselines: Microscaling MXFP4 training INT4 per-tensor training Dampen Freeze Full precision training
Builds on: Microscaling (MX) formats Straight-Through Estimator Dampen (oscillation regularization) Freeze (oscillation freezing) DeiT training recipe
Closest papers by meaning
- ERQ: Error Reduction for Post-Training Quantization of Vision Transformers (ICML 2024 · similarity 0.85)
- FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation (CVPR 2025 · similarity 0.84)
- The Quantization Benefits of Residual-Free Transformers (NeurIPS 2026 · similarity 0.84)
- DiRotQ: Rotation-Aware Quantization for 4-bit Diffusion Transformers (NeurIPS 2026 · similarity 0.84)
- AIQViT: Architecture-Informed Post-Training Quantization for Vision Transformers (AAAI 2025 · similarity 0.84)
- PTQ4DiT: Post-training Quantization for Diffusion Transformers (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.