Research map / Architectures and efficiency / Quantization
Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with hybrid FP8 (1-4-3 forward, 1-5-2 backward) plus round-off residual weight update instead of FP32 | ResNet50 · ImageNet | top-1 accuracy: 76.44 -> 76.22 | No clear effect |
| HFP8 training instead of FP32 | Transformer-base · WMT14 En-De | BLEU: 27.50 -> 27.27 | No clear effect |
| use 1-4-3 FP8 instead of 1-5-2 for inference weights/activations (ablation) | FP32-trained ResNet50 · ImageNet | top-1 accuracy: 75.89 -> 76.46 | Helped |
| use 1-4-3 instead of 1-5-2 for inference (ablation) | SSD-Lite · VOC | mAP: 67.40 -> 68.22 | Helped |
| re-tune BN statistics with 2% of one epoch of data (ablation) | MobileNetV2 FP8 inference · ImageNet | top-1 accuracy | Helped |
| keep depthwise conv layers in FP16 (ablation) | MobileNetV2 FP8 inference · ImageNet | top-1 accuracy | Helped |
| keep last FC output in FP16 (1-6-9) and quantize after max-subtraction in SoftMax (ablation) | HFP8 last layer · WMT En-De / SWB300 | BLEU / WER: ~1 BLEU degradation avoided | Helped |
Closest papers by meaning
- Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks (ICLR 2020 · similarity 0.93)
- Training Deep Neural Networks with 8-bit Floating Point Numbers (NeurIPS 2018 · similarity 0.91)
- Scaling FP8 training to trillion-token LLMs (ICLR 2025 · similarity 0.85)
- µnit Scaling: Simple and Scalable FP8 LLM Training (ICML 2025 · similarity 0.85)
- Towards Fully FP8 GEMM LLM Training at Scale (NeurIPS 2025 · similarity 0.85)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.