Research map / Architectures and efficiency / Quantization
Memory Efficient Optimizers with 4-bit States
Bingrui Li, Jianfei Chen, Jun Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace 32-bit AdamW states with the proposed 4-bit quantized states | AdamW · ImageNet-1k with Swin-T | Accuracy: 81.2 ± 0.05 -> 80.8 ± 0.02 | Hurt |
| Replace 32-bit AdamW states with the proposed 4-bit quantized states | AdamW · E2E-NLG with GPT-2 Medium | BLEU: 67.7 ± 0.67 -> 67.8 ± 0.51 | No clear effect |
| Replace 32-bit AdamW states with the proposed 4-bit quantized states | AdamW · WMT14 en-de with Transformer-Base | SacreBleu: 26.61 ± 0.08 -> 26.28 ± 0.05 | Hurt |
| Use B128/DE rather than B2048/DE for the first moment (ablation) | 4-bit optimizer · ImageNet-1k with Swin-T | Accuracy: 80.9 ± 0.04 -> 81.0 ± 0.06 | Improved |
| Quantize the second moment with Rank-1/Linear (ablation) | 4-bit optimizer · ImageNet-1k with Swin-T | Accuracy: 81.0 ± 0.06 -> 80.8 ± 0.02 | Hurt |
| Factorize the second moment (ablation) | 4-bit optimizer with Rank-1/Linear second moment · ImageNet-1k with Swin-T | Accuracy: 80.8 ± 0.02 -> 80.9 ± 0.06 | Improved |
| Exclude zero from the second-moment dynamic-exponent mapping (ablation) | 4-bit optimizer · E2E-NLG with GPT-2 Medium | BLEU: 66.6 ± 0.61 -> 67.5 ± 0.97 | Improved |
| Use B128/DE rather than B2048/DE for the second moment (ablation) | 4-bit optimizer · E2E-NLG with GPT-2 Medium | BLEU: 66.6 ± 0.61 -> 65.7 ± N/A | Hurt |
| Use rank-1 normalization with linear mapping for the second moment (ablation) | 4-bit optimizer · E2E-NLG with GPT-2 Medium | BLEU: 67.5 ± 0.58 -> 67.8 ± 0.51 | Improved |
| Use the 4-bit optimizer instead of 32-bit AdamW | AdamW · LLaMA-33B instruction tuning | MMLU (5-shot): 56.4 -> 54.9 | Hurt |
| Use the 4-bit optimizer instead of 32-bit AdamW | AdamW · LLaMA-7B training | Total memory: 75.40 GB -> 31.87 GB | Improved |
| Use the 4-bit optimizer instead of 8-bit AdamW | AdamW · RoBERTa-L RTE training | Time: 3.38 min -> 5.59 min | Hurt |
Closest papers by meaning
- 4-bit Shampoo for Memory-Efficient Network Training (NeurIPS 2024 · similarity 0.90)
- LittleBit: Ultra Low-Bit Quantization via Latent Factorization (NeurIPS 2025 · similarity 0.87)
- ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization (NeurIPS 2025 · similarity 0.87)
- RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remapping (NeurIPS 2026 · similarity 0.87)
- BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.87)
- Network Memory Footprint Compression Through Jointly Learnable Codebooks and Mappings (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
