Research map / Architectures and efficiency / Quantization
A2Q+: Improving Accumulator-Aware Weight Quantization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Improves accumulator-aware weight quantization (A2Q) with a zero-centered, sign-agnostic l1-norm bound and Euclidean-projection weight initialization, yielding A2Q+ that gives better accuracy-vs-accumulator-bit-width trade-offs without overflow.
Problem
Quantized neural networks trained for low-precision (e.g., 12-16 bit) accumulators to avoid numerical overflow suffer accuracy loss because the existing A2Q constraint is overly restrictive and its weight initialization adds superfluous quantization error.
Approach
Derive a tighter l1-norm bound (2^P - 2)/(2^N - 1) by zero-centering weights, enforce zero-centering via v - mean(v) with round-toward-zero quantization, and initialize weights by Euclidean projection onto an l1 ball of the pre-trained float weights.
Key results
- On ResNet50 (ImageNet, W4A4), A2Q+ reaches 72.0% top-1 at 12-bit accumulators versus 55.0% for A2Q and 66.7% for A2Q with EP-init, a +17% gain over A2Q.
- A2Q+ maintains over 96% of float accuracy at 14-bit accumulation across ResNet18/34/50 (e.g., 67.1% vs 69.8% baseline range for ResNet18 at P=14 with 69.76% float).
- On CIFAR-10 and BSD300 Pareto frontiers, A2Q+ dominates A2Q and baseline QAT across 14-20 bit accumulators, with the largest gains at extremely low accumulator widths.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use A2Q+ instead of A2Q at P=12 | ResNet18 · ImageNet-1K | top-1 accuracy: 35.5% -> 56.4% | Helped |
| Add EP-init to A2Q at P=12 (ablation) | ResNet18 · ImageNet-1K | top-1 accuracy: 35.5% -> 42.7% | Helped |
| Use A2Q+ instead of A2Q at P=12 | ResNet34 · ImageNet-1K | top-1 accuracy: 43.4% -> 62.1% | Helped |
| Add EP-init to A2Q at P=12 (ablation) | ResNet34 · ImageNet-1K | top-1 accuracy: 43.4% -> 51.4% | Helped |
| Use A2Q+ instead of A2Q at P=12 | ResNet50 · ImageNet-1K | top-1 accuracy: 55.0% -> 72.0% | Helped |
| Add EP-init to A2Q at P=12 (ablation) | ResNet50 · ImageNet-1K | top-1 accuracy: 55.0% -> 66.7% | Helped |
| Use A2Q+ instead of A2Q at P=14 | ResNet50 · ImageNet-1K | top-1 accuracy: 73.8% -> 75.7% | Helped |
| Add EP-init to A2Q at P=14 (ablation) | ResNet50 · ImageNet-1K | top-1 accuracy: 73.8% -> 74.5% | Helped |
| Add EP-init to A2Q at P=16 (ablation) | ResNet50 · ImageNet-1K | top-1 accuracy: 76.0% -> 76.0% | No clear effect |
| Use A2Q+ instead of A2Q at P=14 | ResNet18 · ImageNet-1K | top-1 accuracy: 60.3% -> 67.1% | Helped |
| Use A2Q+ instead of A2Q at P=14 | ResNet34 · ImageNet-1K | top-1 accuracy: 65.8% -> 71.4% | Helped |
| Use A2Q+ instead of A2Q at P=16 | ResNet18 · ImageNet-1K | top-1 accuracy: 69.2% -> 69.8% | Helped |
Surprising findings
- Zero-centering weights introduces implicit dimensionality reduction that harms depthwise separable convolutions with small dot-product sizes, so the authors use A2Q for depthwise and A2Q+ for pointwise layers.
- Only 52% of ResNet18 output channels in the PyTorch checkpoint satisfy a 16-bit accumulator constraint under A2Q, and 23% satisfy 14-bit, indicating naive initialization is severely constrained.
Limitations
- A2Q+ is initialized with the A2Q initialization rather than the Euclidean projection, because the projection does not directly apply under zero-centering.
- Post-training quantization performance is not improved by the better initialization, due to round-to-zero and severe accumulator constraints.
- Zero-centering causes implicit dimensionality reduction that negatively affects depthwise convolutions; results rely on uniform-precision quantization design spaces and PyTorch checkpoints.
Experimental setup
Datasets: CIFAR-10 BSD300 ImageNet-1K
Benchmarks: CIFAR-10 BSD300 ImageNet-1K
Metrics: test top-1 accuracy PSNR weight sparsity initial weight quantization error
Baselines: A2Q Baseline QAT Exhaustive bit width manipulation
Builds on: A2Q weight normalization Brevitas FINN straight-through estimator
Closest papers by meaning
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators (ICLR 2024 · similarity 0.90)
- HAWQ-V3: Dyadic Neural Network Quantization (ICML 2021 · similarity 0.88)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.87)
- MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization (NeurIPS 2024 · similarity 0.87)
- Analog Foundation Models (NeurIPS 2025 · similarity 0.87)
- Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.