Research map / Architectures and efficiency / Quantization
Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes weight bias correction (sharing BN across bit-widths without calibration) and bit-wise gradient-based coreset sampling to cut the training cost of multi-bit quantization networks.
Problem
Multi-bit quantization networks require full-dataset updates per trained bit-width and extra calibration or fine-tuning to support additional precisions, giving high training overhead.
Approach
Correct quantization-induced shift and scale bias in the quantized weights so that child models at different bit-widths share BN layers (with a separate BN for 1-bit) and need no calibration; then select per-bit-width coresets using TDDS-style gradient/training-dynamics scores computed under a bit-wise training schedule and resampled periodically.
Key results
- Reduces training time up to 7.88x while keeping competitive or superior per-bit accuracy on CIFAR-10/100, TinyImageNet and ImageNet-1K.
- On ImageNet-1K ResNet-50, the method reduces training time 5.75x relative to dedicated training with minimal accuracy loss.
- Bias correction reduces average activation MAE between quantized and full-precision models from 0.755 to 0.667.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add weight bias correction and bit-wise coreset sampling | Multi-bit quantization networks · CIFAR-10/100, TinyImageNet, ImageNet-1K | training time: up to 7.88× | Helped |
| Use the proposed method instead of dedicated training | ResNet-50 · ImageNet-1K | training time: 5.75× | Helped |
| Add bit-wise coreset sampling | DeiT-T · ImageNet-1K | GPU hours: 18.22 GPU hours reduction | Helped |
| Add bit-wise coreset sampling | DeiT-S · TinyImageNet | GPU hours: 8.41× | Helped |
| Add bias correction (ablation) | Quantized models · Activation alignment, averaged across bit-widths | MAE: 0.755 to 0.667 | Helped |
| Add BN adaptation after bias correction (ablation) | Multi-bit quantization networks · Accuracy across bit-widths | accuracy | Helped |
| Use bit-wise rather than batch-wise TDDS scores for coreset selection (ablation) | Any-Precision with Bias Correction · CIFAR-10 and CIFAR-100 at different pruning ratios | accuracy | Helped |
| Resample the coreset every epoch rather than every 30 epochs (ablation) | PreActResNet-20 · CIFAR-10 | average accuracy: +1.33%p | Helped |
| Share BN across all bit-widths, including 1-bit, rather than assigning 1-bit its own BN (ablation) | PreActResNet-20 · CIFAR-10, 1-bit quantization | accuracy | Hurt |
| Use coreset sampling alone without bias correction and BN adaptation (ablation) | Multi-bit quantization networks · CIFAR-10, accuracy across bit-widths | accuracy | Hurt |
| Use a fixed temperature of 0.5 rather than fixed temperatures 0.1 or 1.0 (ablation) | PreActResNet-20 · CIFAR-10 | test accuracy | Helped |
| Dynamically increase sampling temperature from 0.1 to 1.0 rather than use fixed 0.5 (ablation) | PreActResNet-20 · CIFAR-10 | test accuracy | Hurt |
Surprising findings
- Assigning a separate BN layer to 1-bit while sharing BN across other bit-widths is necessary, since sharing BN including 1-bit significantly degrades 1-bit accuracy.
- Dynamic score re-evaluation gives only marginal accuracy gains while greatly increasing cost; one-time scoring often matches or beats it.
Limitations
- Evaluation limited to computer vision tasks; generative and language tasks not explored.
- 1-bit weights break the bias correction assumption, requiring a separate BN layer for 1-bit.
- Dynamic score re-evaluation is costly, so importance scores are computed once and held fixed.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 TinyImageNet ImageNet-1K
Metrics: per-bit-width accuracy GPU hours
Baselines: Dedicated training Any-Precision Random Entropy Forgetting EL2N Moderate TDDS
Builds on: Any-Precision TDDS DoReFa StatsQ OFQ
Closest papers by meaning
- DQT: Dynamic Quantization Training via Dequantization-Free Nested Integer Arithmetic (AAAI 2026 · similarity 0.90)
- Scalable methods for 8-bit training of neural networks (NeurIPS 2018 · similarity 0.90)
- AMPA: Adaptive Mixed Precision Allocation for Low-Bit Integer Training (ICML 2024 · similarity 0.89)
- Retraining-Free Model Quantization via One-Shot Weight-Coupling Learning (CVPR 2024 · similarity 0.87)
- PTMQ: Post-training Multi-Bit Quantization of Neural Networks (AAAI 2024 · similarity 0.87)
- QBB: Quantization with Binary Bases for LLMs (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.