Research map / Architectures and efficiency / Quantization
Training with Quantization Noise for Extreme Model Compression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with Quant-Noise before iPQ instead of using iPQ alone | 16-layer Transformer · WikiText-103 test | perplexity: 25.2 -> 20.7 | Helped |
| Train with Quant-Noise before iPQ instead of using iPQ alone | EfficientNet-B3 · ImageNet-1k validation | top-1 accuracy: 79.0 -> 80.0 | Helped |
| Use QAT rather than partial random-weight quantization during iPQ training | 16-layer Transformer · WikiText-103 test | perplexity: 20.7 -> 41.2 | Hurt |
| Use QAT rather than partial random-weight quantization during iPQ training | EfficientNet-B3 · ImageNet-1k validation | top-1 accuracy: 80.0 -> 55.7 | Hurt |
| Train with Quant-Noise before int4 instead of post-training int4 quantization | 16-layer Transformer · WikiText-103 test | perplexity: 39.4 -> 21.8 | Helped |
| Train with Quant-Noise before int4 instead of post-training int4 quantization | EfficientNet-B3 · ImageNet-1k validation | top-1 accuracy: 45.3 -> 67.8 | Helped |
| Train with Quant-Noise before int8 instead of post-training int8 quantization | 16-layer Transformer · WikiText-103 test | perplexity: 19.6 -> 18.7 | Helped |
| Train with Quant-Noise before int8 instead of post-training int8 quantization | EfficientNet-B3 · ImageNet-1k validation | top-1 accuracy: 80.7 -> 80.9 | Helped |
| Add int8 quantization of centroids and activations to iPQ + Quant-Noise | 16-layer Transformer · WikiText-103 test | perplexity: 20.7 -> 21.1 | Hurt |
| Replace exact PQ noise with proxy zeroing noise (ablation) | iPQ-trained 16-layer Transformer · WikiText-103 test | quantized perplexity: 21.1 -> 21.0 | No clear effect |
| Select noise blocks by clusters rather than all subvectors (ablation) | iPQ-trained 16-layer Transformer · WikiText-103 test | quantized perplexity: 21.1 -> 21.1 | No clear effect |
| Apply STE to LayerDrop pruning noise (ablation) | Quant-Noise + Share + Prune · WikiText-103 test | perplexity: 24.2 -> 24.5 | Hurt |
Closest papers by meaning
- Effective Interplay between Sparsity and Quantization: From Theory to Practice (ICLR 2025 · similarity 0.88)
- PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression (NeurIPS 2024 · similarity 0.88)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.88)
- SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression (ICLR 2024 · similarity 0.88)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.87)
- Extreme Compression of Large Language Models via Additive Quantization (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.