Research map / Architectures and efficiency / Quantization
PikeLPN: Mitigating Overlooked Inefficiencies of Low-Precision Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends the ACE arithmetic cost metric to elementwise operations (ACEv2) and designs PikeLPN, a MobileNetV1-based low-precision network that quantizes batch norm, activations and quantization scales, reaching Pareto-better accuracy-efficiency on ImageNet.
Problem
Efficiency metrics for low-precision (≤4-bit/binary) networks such as ACE and CPU64 ignore non-quantized elementwise operations (batch norm, parameterized activations, quantization scaling), which dominate inference arithmetic cost of these models.
Approach
Extend ACE to ACEv2 by adding cost formulas for elementwise multiplication, addition and shift operations, then use it to guide a separable-convolution low-precision architecture that quantizes elementwise ops too via QuantNorm, Double Quantization and Distribution-Heterogeneous Quantization.
Key results
- PikeLPN-1x reaches 67.55% Top-1 on ImageNet with 8.50e9 ACEv2 and 34.98 mJ arithmetic energy, versus PokeBNN-0.5x at 65.2% with 33.58e9 ACEv2 and 143.78 mJ.
- PikeLPN-3x achieves 71.95% Top-1 with 33.70e9 ACEv2, 1.5% above PokeBNN-0.75x at 70.5% while being about 35% more efficient in ACEv2 terms.
- Elementwise operations drop to under 5% of arithmetic energy in PikeLPN, versus 31-93% in prior low-precision models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PoT-4 for pointwise and linear-8 for depthwise convolution weights instead of PoT for both (ablation) | PikeLPN separable convolutions · ImageNet | Top-1 accuracy: 64.50 -> 67.60 | Helped |
| use PoT-8 quantization parameters for the linear-8 depthwise quantizer (ablation) | PikeLPN separable convolutions · ImageNet | Top-1 accuracy: 67.60 -> 67.55 | No clear effect |
| use PoT-8 quantization parameters for the linear-8 depthwise quantizer (ablation) | PikeLPN separable convolutions · ImageNet | ACEv2: 12.86 -> 10.95 | Helped |
| use QuantNorm instead of folded batch normalization (ablation) | PikeLPN · ImageNet | validation Top-1 accuracy | Helped |
| use QuantNorm instead of vanilla 8-bit batch normalization (ablation) | PikeLPN · ImageNet | validation Top-1 accuracy | Helped |
| replace ReLU with PReLU (ablation) | 4-bit MobileNetV2 · ImageNet | ACEv2: 20.44 -> 26.5 | Hurt |
| replace ReLU with DPReLU (ablation) | 4-bit MobileNetV2 · ImageNet | ACEv2: 20.44 -> 27.67 | Hurt |
| use sub-channelwise instead of channelwise quantization (ablation) | MobileNetV2 (4W, 4A) · ImageNet | ACEv2: 20.44 -> 27.06 | Hurt |
| use sub-channelwise instead of channelwise quantization (ablation) | ResNet50 (1W, 1A) · ImageNet | ACEv2: 28.13 -> 50.08 | Hurt |
| increase the number of parallel branches (ablation) | ResNet-50 · ResNet-50 | Arithmetic Intensity: 73.5 -> 49.66 -> 36.75 | Hurt |
| use distribution-heterogeneous quantization (PoT pointwise, linear depthwise) instead of PoT for both separable-convolution layers (ablation) | PikeLPN · ImageNet | ACEv2: 10.05 -> 12.86 | Hurt |
Surprising findings
- Non-quantized elementwise operations account for up to 89% of arithmetic energy in binary models, contrary to the common assumption that MAC operations dominate cost.
- Parameterized activations PReLU and DPReLU increase the ACEv2 cost of a 4-bit MobileNetV2 by 29.6% and 35.3%, respectively.
Limitations
- Cost metrics are hardware-agnostic estimates based on 45nm CMOS energy values rather than measured hardware latency.
- Evaluation is limited to ImageNet image classification with MobileNetV1-style architecture.
Experimental setup
Datasets: ImageNet (ILSVRC12)
Benchmarks: ImageNet
Metrics: Top-1 accuracy ACEv2 arithmetic energy (mJ) arithmetic intensity
Baselines: PokeBNN-0.5x PokeBNN-0.75x PokeBNN-1x ReActNet PROFIT MobiNet MeliusNet-29 MeliusNet-42 Bi-RealNet-18 Bi-RealNet-34 XNOR-Net Real-to-Binary Net
Builds on: ACE MobileNetV1 PokeBNN QKeras quantization-aware training PReLU
Closest papers by meaning
- Scalable methods for 8-bit training of neural networks (NeurIPS 2018 · similarity 0.87)
- Not All Bits have Equal Value: Heterogeneous Precisions via Trainable Noise (NeurIPS 2022 · similarity 0.87)
- HAWQ-V3: Dyadic Neural Network Quantization (ICML 2021 · similarity 0.87)
- Towards Cheaper Inference in Deep Networks with Lower Bit-Width Accumulators (ICLR 2024 · similarity 0.86)
- MagR: Weight Magnitude Reduction for Enhancing Post-Training Quantization (NeurIPS 2024 · similarity 0.86)
- LOTION: Smoothing the Optimization Landscape for Quantized Training (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.