Research map / Architectures and efficiency / Quantization
Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LUQ FP4 neural gradients with INT4 weights and activations instead of Ultra-low radix-4 training | ResNet-50 · ImageNet | Top-1 accuracy: 74.01% -> 75.42 % | Helped |
| Add two stochastic gradient-quantization samples and average (ablation) | LUQ · ImageNet | Top-1 accuracy: 75.42 % -> 75.63 % | Helped |
| Add high-precision fine-tuning after LUQ + SMP (ablation) | ResNet-50 · ImageNet | Top-1 accuracy: 75.63% -> 76.18 % | Helped |
| Add stochastic gradient-quantization samples and average (ablation) | LUQ · WMT En-De | BLEU: 27.17 -> 27.25 | Helped |
| Add stochastic gradient-quantization samples and average (ablation) | LUQ · SQUAD | F1: 85.75 -> 85.9 | Helped |
| Use the Hindsight maximum estimate instead of measuring the current maximum (ablation) | LUQ · ImageNet | Top-1 accuracy: 75.42 -> 75.4 | No clear effect |
| Replace standard LUQ scaling with power-of-two LUQ (ablation) | LUQ · ImageNet | Top-1 accuracy: 75.42 % -> 75.15 % | Hurt |
| Quantize the backward phase to FP4 rather than keeping it in FP32 (ablation) | INT4-quantized forward phase · ImageNet | Top-1 accuracy: 76.35 % -> 75.4 % | Hurt |
| Quantize the forward phase to INT4 rather than keeping it in FP32 (ablation) | FP4-quantized backward phase · ImageNet | Top-1 accuracy: 75.6 % -> 75.4 % | Hurt |
Closest papers by meaning
- Post training 4-bit quantization of convolutional networks for rapid-deployment (NeurIPS 2019 · similarity 0.92)
- Ultra-Low Precision 4-bit Training of Deep Neural Networks (NeurIPS 2020 · similarity 0.91)
- Training Transformers with 4-bit Integers (NeurIPS 2023 · similarity 0.91)
- Scalable methods for 8-bit training of neural networks (NeurIPS 2018 · similarity 0.90)
- A Statistical Framework for Low-bitwidth Training of Deep Neural Networks (NeurIPS 2020 · similarity 0.90)
- Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.