Research map / Architectures and efficiency / Quantization
You Already Have It: A Generator-Free Low-Precision DNN Training Framework Using Stochastic Rounding
Geng Yuan, Sung-En Chang, Qing Jin, Alec Lu, Yanyu Li, Yushu Wu, Zhenglun Kong, Yanyue Xie, Peiyan Dong, Minghai Qin, Xiaolong Ma, Xulong Tang and 2 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use gradients as the random-number source instead of errors (ablation) | generator-free stochastic rounding · CIFAR-100 (ResNet-32) 6-bit | accuracy (%): 8.34 -> 74.06 | Improved |
| use gradients instead of activations as source (ablation) | generator-free stochastic rounding · CIFAR-100 (ResNet-32) 6-bit | accuracy (%): 70.19 -> 74.06 | Improved |
| use gradients instead of weights as source (ablation) | generator-free stochastic rounding · CIFAR-100 (ResNet-32) 6-bit | accuracy (%): 73.32 -> 74.06 | Improved |
| use gradients instead of errors as source (ablation) | generator-free stochastic rounding · CIFAR-10 (ResNet-20) 6-bit | accuracy (%): 71.11 -> 91.88 | Improved |
| use gradients instead of activations as source (ablation) | generator-free stochastic rounding · CIFAR-10 (ResNet-20) 6-bit | accuracy (%): 90.36 -> 91.88 | Improved |
| extract from later layers of errors (layer-31 vs layer-2) (ablation) | generator-free stochastic rounding · CIFAR-100 (ResNet-32) 6-bit | accuracy (%): 8.34 -> 54.22 | Improved |
| synthesize random numbers from LSBs of multiple source data instead of direct bit extraction (ablation) | generator-free stochastic rounding · ResNet-20 CIFAR-10 | random number quality | Improved |
| add number-mapping strategy to biased sources (ablation) | generator-free stochastic rounding · biased sources (activations, errors) | accuracy | Improved |
| source with higher information entropy (ablation) | generator-free stochastic rounding · CIFAR-100 (ResNet-32) 6-bit | information entropy (bits): 2.8017 -> 5.8774 | Improved |
| replace LFSR-based RNGs with generator-free extraction | low-precision training · ImageNet ResNet-18 | accuracy (%): 70.72 -> 71.10 | Improved |
| replace LFSR RNGs with extraction | low-precision training · CIFAR-10/100 6-bit and 8-bit | accuracy (%): 91.83 -> 91.88 | No clear effect |
| replace stochastic rounding with nearest rounding | low-precision training · ImageNet ResNet-18 | accuracy (%): 71.10 -> 68.13 | Hurt |
Closest papers by meaning
- Training Deep Neural Networks with 8-bit Floating Point Numbers (NeurIPS 2018 · similarity 0.85)
- Improving Neural Network Training in Low Dimensional Random Bases (NeurIPS 2020 · similarity 0.85)
- Mixed Precision Training (ICLR 2018 · similarity 0.85)
- CPT: Efficient Deep Neural Network Training via Cyclic Precision (ICLR 2021 · similarity 0.84)
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators (ICLR 2024 · similarity 0.82)
- Partially Stochastic Infinitely Deep Bayesian Neural Networks (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
