Research map / Architectures and efficiency / Quantization
Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that DNNs pretrained in full precision can be fine-tuned to run with 12-bit floating-point accumulators (M7E4) with little accuracy loss, and that fine-grained straight-through gradient estimators help at lower accumulator precision.
Problem
Accumulation of products in GEMM operations is still done at high precision in hardware, and it dominates cost once weights and activations are FP8; low-bit accumulators usually degrade accuracy badly.
Approach
Simulate a quantized FMA (product and accumulator quantized with floating-point rounding and overflow/underflow) in CUDA, fine-tune pretrained networks with underflow disabled and then enabled in a two-stage schedule, and for sub-12-bit accumulators replace the identity straight-through estimator with fine-grained STEs (Recursive/OF, Immediate/DIFF) that backpropagate through the accumulation grap
Key results
- ResNet50 on ImageNet with 8-bit FP weights/activations and 12-bit (M7E4) accumulators reaches 76.22% top-1 with dual-stage fine-tuning vs. 76.25% for the FP8 baseline with 32-bit accumulators.
- On MNIST, an M4E3 (8-bit) accumulator fails with identity STE (18.28%) but reaches 98.47% with Immediate/OF and Recursive/OF STEs, close to the 98.65% baseline.
- On LLaMA-2-7B fine-tuned with QLoRA, an M7E4 accumulator gives 45.1 MMLU 5-shot vs. 45.3 baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 12-bit accumulators with dual-stage fine-tuning | ResNet50 · ImageNet | Top-1 accuracy: 76.80% -> 76.40% | No clear effect |
| Use dual-stage rather than 1-stage fine-tuning (ablation) | LBA ResNet50 with FP8 weights and activations · ImageNet | Top-1 accuracy: 74.15% -> 76.22% | Helped |
| Use dual-stage rather than 1-stage fine-tuning (ablation) | LBA ResNet18 with FP8 weights and activations · ImageNet | Top-1 accuracy: 69.54% -> 69.70% | Helped |
| Use dual-stage rather than 1-stage fine-tuning (ablation) | LBA ResNet34 with FP8 weights and activations · ImageNet | Top-1 accuracy: 73.18% -> 73.42% | Helped |
| Use LBA accumulators instead of the baseline accumulator | Bert-Large · SQuAD v1 | f1: 90.40 -> 90.66 | Helped |
| Reduce accumulator precision from M10E5 to M6E5 (ablation) | LLaMA v2 7B · MMLU 5-shot | test accuracy: 45.4 -> 44.3 | Hurt |
| Use M7E4 rather than M10E5 accumulator (ablation) | LLaMA v2 7B · MMLU 5-shot | test accuracy: 45.4 -> 45.1 | Hurt |
| Replace identity STE with Immediate / DIFF STE (ablation) | Fully connected NN with M4E3 accumulators · MNIST | Top-1 accuracy: 18.28 -> 97.67 | Helped |
| Replace identity STE with Immediate / DIFF STE (ablation) | LBA transformer, M3E3 · Masked Language Modeling | Accuracy: 20.86 -> 24.60 | Helped |
| Replace identity STE with Immediate / DIFF STE (ablation) | LBA transformer, M4E3 · Masked Language Modeling | Accuracy: 13.88 -> 41.94 | Helped |
| Replace identity STE with Immediate / DIFF STE (ablation) | LBA transformer, M5E3 · Masked Language Modeling | Accuracy: 9.47 -> 50.12 | Helped |
| Replace identity STE with Immediate / DIFF STE (ablation) | LBA transformer, M6E3 · Masked Language Modeling | Accuracy: 14.71 -> 50.03 | Helped |
Limitations
- Backpropagation through the full accumulation graph requires recomputation, roughly doubling training time, and is only feasible for fully connected and attention layers, not convolutions.
- Fine-grained STEs did not fully close the gap to baseline at extreme accumulator precision (e.g., M3E3 on masked LM reached at most 24.60% vs 51.31% FP32).
- Gate-count hardware analysis is a first-order estimate; accumulation is simulated rather than implemented in hardware.
Experimental setup
Datasets: ImageNet SQuAD v1 OASST1 MNIST oscar unshuffled-original-af CIFAR10
Benchmarks: ImageNet SQuAD v1 MMLU MNIST
Metrics: top-1 accuracy exact match F1 5-shot accuracy
Baselines: Baseline FP32 accumulator Baseline (FP8) with 32-bit accumulator Sakr et al. 16-bit accumulator WrapNet Identity STE
Builds on: WrapNet chunk-based accumulation qtorch QLoRA
Closest papers by meaning
- A2Q+: Improving Accumulator-Aware Weight Quantization (ICML 2024 · similarity 0.90)
- Accumulation Bit-Width Scaling For Ultra-Low Precision Training Of Deep Networks (ICLR 2019 · similarity 0.89)
- Not All Bits have Equal Value: Heterogeneous Precisions via Trainable Noise (NeurIPS 2022 · similarity 0.89)
- Don’t Waste Your Bits! Squeeze Activations and Gradients for Deep Neural Networks via TinyScript (ICML 2020 · similarity 0.88)
- Linear Symmetric Quantization of Neural Networks for Low-precision Integer Hardware (ICLR 2020 · similarity 0.88)
- Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.