Research map / Architectures and efficiency / Quantization

AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer Training

ICML 2024 · Architectures and efficiency › Quantization · approach: Low-precision training formats

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AMPA, an integer low-bit training framework that adaptively allocates INT4/INT6/INT8 bitwidths layer-wise to weights, activations, and gradients using a magnitude-based sensitivity measure, cutting BitOPs ~35-49% versus INT8 training.

Problem

Low-bit integer training is commonly limited to INT8, and existing mixed-precision methods (solution-space search, Hessian-based sensitivity) are impractical or ineffective for training, where gradients are non-zero and bitwidth impact changes over training.

Approach

A magnitude-based layer-wise sensitivity measure (product of expected gradient magnitude and expected quantization deviation, with an upper-bound justification via Cauchy-Schwarz) drives periodic bitwidth increases from INT4 toward INT8, with a "taboo list" threshold to ensure fair allocation across layers.

Key results

• On CIFAR-10, AMPA reaches 92.26% for ResNet-20 vs 91.95% for UI8 with 41.59% BitOPs reduction; on ImageNet ResNet-18 it gets 69.58% vs 69.67% for UI8 with 47.40% reduction.
• On ImageNet ResNet-50 AMPA achieves 74.94% vs 76.34% for UI8 at 40.71% BitOPs reduction, a loss larger than 2%.
• Simulated FPGA training of ResNet-20 shows 19.16% latency reduction compared with INT8 training.

Measured effects reported in the paper

Surprising findings

• Hessian-trace-based sensitivity yields higher compression but significantly worse accuracy for weight and activation bitwidth selection than the proposed magnitude-based measure.
• Setting the layer update ratio with larger weight ratio than gradient ratio (alpha > beta > gamma) harms performance, contrary to the intuition that gradients are less sensitive.

Limitations

• Accuracy loss on some models exceeds the claimed 2% (e.g., ResNet-50 ImageNet 1.4 points, ViT-S CIFAR/ImageNet results mixed), and some configurations diverge (NaN) with high update ratios or no fair-allocation threshold.
• Hyperparameters (update interval, layer update ratios, fair-allocation threshold) require tuning per task and affect compression vs accuracy trade-offs.
• Acceleration evaluated only via FPGA simulation on ResNet-20; BitOPs is a proxy rather than measured end-to-end speed on hardware.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 ImageNet Dsb2018 Kvasir Wikitext-2 Wikitext-103 Penn Treebank

Benchmarks: CIFAR-10 CIFAR-100 ImageNet Dsb2018 Kvasir Wikitext-2 Wikitext-103 Penn Treebank

Metrics: top-1 accuracy BitOPs reduction ratio Dice similarity coefficient perplexity

Baselines: UI8 DAI8 WAGEUBN Hessian trace sensitivity distribution-based measurement (Zhang et al. 2020) full INT6 training

Builds on: UI8 (Unified INT8 training) DAI8 (distributive adaptive INT8) Zhang et al. 2020 distribution-based mixed-precision gradients Hessian-based sensitivity (HAWQ-style, Dong et al.) BitOPs metric (Yang & Jin 2021)

Closest papers by meaning

• Not All Bits have Equal Value: Heterogeneous Precisions via Trainable Noise (NeurIPS 2022 · similarity 0.89)
• Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling (NeurIPS 2025 · similarity 0.89)
• MetaMix: Meta-State Precision Searcher for Mixed-Precision Activation Quantization (AAAI 2024 · similarity 0.88)
• DQT: Dynamic Quantization Training via Dequantization-Free Nested Integer Arithmetic (AAAI 2026 · similarity 0.88)
• Training Quantized Neural Networks With a Full-Precision Auxiliary Module (CVPR 2020 · similarity 0.88)
• Retraining-Free Model Quantization via One-Shot Weight-Coupling Learning (CVPR 2024 · similarity 0.86)