Research map / Architectures and efficiency / Quantization
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
Jianfei Chen, Lianmin Zheng, Zhewei Yao, Dequan Wang, Ion Stoica, Michael Mahoney, Joseph Gonzalez
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ActNN (L3) activation compression at 2 bits instead of full-precision activations | ResNet-50 · ImageNet | top-1 accuracy: 77.1 -> 76.9 | No clear effect |
| Replace BLPA per-tensor quantization with ActNN (L3) mixed-precision quantization at 1.5 bits | ResNet-50 · ImageNet | top-1 accuracy: 76.6 -> 76.4 | Mixed |
| Allocate bits across layers as well as samples (L3 rather than L2.5) (ablation) | ActNN quantization · ImageNet, ResNet-50 | top-1 accuracy: 75.9 -> 76.4 | Improved |
| Allocate bits between samples while keeping layer precision equal (L2.5 instead of L2) (ablation) | ActNN quantization · ImageNet, ResNet-50 at 2 bits | top-1 accuracy: 0.1 -> 77.1 | No clear effect |
| Combine ActNN (L3) activation compression with AMP | ResNet-50 · ImageNet | validation accuracy: 77.3 -> 76.9 | No clear effect |
| Increase the batch size from 8 to 160 with ActNN (L3) | Dilation8 · Semantic segmentation | mIoU: 72.92 -> 75.85 | Improved |
| Increase the batch size from 8 to 160 with ActNN (L3) | FPN · Semantic segmentation | mIoU: 74.18 -> 75.98 | Improved |
| Use ActNN (L3), 2-bit activations, and a larger batch | MoCov2 · ImageNet | validation accuracy: 67.69 -> 67.25 | No clear effect |
| Use ActNN (L3), 2-bit activations, and a larger batch | BYOL · ImageNet | validation accuracy: 72.35 -> 72.65 | Improved |
| Estimate gradient magnitude using moving averages rather than stale gradients (ablation) | ActNN (L3) · Quantization strategy ablation | accuracy | No clear effect |
Closest papers by meaning
- Accelerating Convolutional Neural Networks via Activation Map Compression (CVPR 2019 · similarity 0.87)
- Dynamic Network Quantization for Efficient Video Inference (ICCV 2021 · similarity 0.87)
- GACT: Activation Compressed Training for Generic Network Architectures (ICML 2022 · similarity 0.87)
- Few-bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reduction (ICML 2023 · similarity 0.87)
- Don’t Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript (ICML 2020 · similarity 0.87)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
