Research map / Architectures and efficiency / Quantization
Training Deep Neural Networks with 8-bit Floating Point Numbers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with FP8 representations and multiplications, FP16 accumulation and stochastic-rounded FP16 updates | CNN and DNN models · CIFAR10, BN50, ImageNet | test error: 17.80% -> 18.15%; 7.23% -> 7.79%; 59.33% -> 60.08%; 41.96% -> 42.45%; 32.57% -> 33.05%; 27.86% -> 28.28% | Mixed |
| Use FP8 for all AlexNet training operands with FP16 accumulation | AlexNet · ImageNet | top-1 accuracy: 57.5 vs 56.9 | Helped |
| Add chunk-based accumulation to FP16 GEMM accumulation (ablation) | FP8 training · ResNet50 | convergence | Helped |
| Use FP32 for Gradient GEMM instead of reduced-precision accumulation without chunking (ablation) | ResNet18 FP8 training · ImageNet | test error: 99% | Helped |
| Use FP16 input images instead of FP8 (ablation) | ResNet18 and ResNet50 FP8 training · ImageNet | accuracy: ~2% | Helped |
| Use FP16 rather than FP8 for all last-layer GEMMs and Softmax input (ablation) | AlexNet · ImageNet | test error: 52.12 -> 42.30 | Helped |
| Preserve the final forward GEMM output in FP16 while keeping last-layer GEMMs in FP8 (ablation) | AlexNet · ImageNet | test error: 52.12 -> 42.37 | Helped |
| Use stochastic instead of nearest rounding for FP16 weight updates (ablation) | AlexNet · ImageNet | top-1 accuracy: 54.10% -> 57.94% | Helped |
| Use stochastic instead of nearest rounding for FP16 weight updates (ablation) | ResNet18 · ImageNet | top-1 accuracy: 65.74% -> 67.34% | Helped |
| Use chunk sizes between 64 and 256 for Gradient GEMM (ablation) | CIFAR10-ResNet · CIFAR10 | normalized L2-distance to FP32 GEMM: 64 and 256 | Helped |
| Replace chunk-based accumulation with ordinary FP16 accumulation using nearest rounding (ablation) | FP8 training dot products · Numerical accumulation test | accumulation fidelity: length≥4096 | Hurt |
Closest papers by meaning
- Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (ICLR 2020 · similarity 0.92)
- Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks (NeurIPS 2019 · similarity 0.91)
- Mixed Precision Training (ICLR 2018 · similarity 0.89)
- A Block Minifloat Representation for Training Deep Neural Networks (ICLR 2021 · similarity 0.89)
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators (ICLR 2024 · similarity 0.85)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.