Research map / Architectures and efficiency / Quantization
Neural gradients are near-lognormal: improved quantized and sparse training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the lognormal-optimized FP6 format instead of FP32 gradients | ResNet18 · ImageNet val | top-1 accuracy: 70.4% -> 70.0% | No clear effect |
| Use the lognormal-optimized FP7 format instead of FP32 gradients | ResNet18 · ImageNet val | top-1 accuracy: 70.4% -> 70.4% | No clear effect |
| Use optimized FP6 gradients instead of FP32 gradients | ResNet18 · Cifar100 val | accuracy: 64.9% -> 64.9% | No clear effect |
| Use optimized FP5 gradients instead of FP32 gradients | ResNet101 · Cifar100 val | accuracy: 71.3% -> 70.4% | Hurt |
| Use per-layer gradient scaling instead of global dynamic loss scaling | ResNet18 with FP4 (1-3-0) · ImageNet val | top-1 accuracy: 3% -> 64.8% | Helped |
| Use per-layer gradient scaling instead of static global scaling at 2^18 | ResNet18 with FP4 (1-3-0) · ImageNet val | top-1 accuracy: 54.9% -> 64.8% | Helped |
| Choose the mantissa/exponent partition using a normal rather than lognormal assumption (ablation) | FP gradient quantization · ImageNet and Cifar100 | validation accuracy | Hurt |
| Allocate different sparsity levels by layer rather than using homogeneous sparsity (ablation) | Stochastic gradient pruning · ImageNet val | gradient sparsity: 80% -> 85% | Helped |
| Increase heterogeneous pruning toward nearly 90% sparsity (ablation) | ResNet18 · ImageNet val | accuracy degradation: less than 0.3% | Hurt |
| Use lognormal rather than normal gradient magnitudes to fit the distribution (ablation) | BERT · CoLA | Kolmogorov-Smirnov distance: 0.46$\pm$0.02 -> 0.05$\pm$0.002 | Helped |
| Use lognormal-derived pruning thresholds instead of thresholds based on a normal assumption (ablation) | Stochastic gradient pruning · ImageNet and other reported models | required-versus-achieved sparsity accuracy | Helped |
Closest papers by meaning
- Minimum Variance Unbiased N:M Sparsity for the Neural Gradients (ICLR 2023 · similarity 0.89)
- Accurate Neural Training with 4-bit Matrix Multiplications at Standard Formats (ICLR 2023 · similarity 0.89)
- Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! (ICLR 2023 · similarity 0.88)
- Robust Training of Neural Networks at Arbitrary Precision and Sparsity (ICLR 2026 · similarity 0.87)
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.87)
- Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.