Research map / Architectures and efficiency / Quantization
Memory-Efficient 4-bit Preconditioned Stochastic Optimization
Jingyang Li, Kuangyu Ding, Kim-Chuan Toh, Pan Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Cholesky quantization instead of vanilla quantization (ablation) | 4-bit Shampoo · CIFAR-100, ViT-Small (AdamW) | test accuracy (%): 74.56 -> 77.34 | Improved |
| add Cholesky quantization instead of vanilla quantization (ablation) | 4-bit Shampoo · CIFAR-100, ResNet-34 (SGDM) | test accuracy (%): 79.45 -> 80.27 | Improved |
| add Cholesky quantization instead of vanilla quantization (ablation) | 4-bit Shampoo · CIFAR-100, VGG-19 (SGDM) | test accuracy (%): 74.36 -> 74.99 | Improved |
| add error feedback to Cholesky quantization (ablation) | 4-bit Shampoo (CQ) · CIFAR-100, ResNet-34 (SGDM) | test accuracy (%): 80.27 -> 80.52 | Improved |
| add error feedback to Cholesky quantization (ablation) | 4-bit Shampoo (CQ) · CIFAR-100, ViT-Small (AdamW) | test accuracy (%): 77.34 -> 77.74 | Improved |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · Tiny-ImageNet, ViT-Small (AdamW) | test accuracy (%): 56.47 -> 57.51 | Mixed |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · Tiny-ImageNet, ResNet-34 (SGDM) | test accuracy (%): 68.43 -> 68.88 | Improved |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · ImageNet, ViT-Base (AdamW) | test accuracy (%): 72.28 -> 75.01 | Improved |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · ImageNet, ResNet-50 (SGDM) | test accuracy (%): 77.73 -> 78.00 | Improved |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · C4, LLaMA-130M | test perplexity: 28.08 -> 26.98 | Improved |
| add Cholesky quantization with error feedback instead of vanilla (ablation) | 4-bit Shampoo · C4, LLaMA-1B | test perplexity: 48.53 -> 46.31 | Improved |
| quantize only off-diagonal entries, keep diagonal in 32-bit (ablation) | vanilla 4-bit Shampoo · CIFAR-100, VGG-19 | test accuracy (%): 74.20 -> 74.36 | Improved |
Closest papers by meaning
- 4-bit Shampoo for Memory-Efficient Network Training (NeurIPS 2024 · similarity 0.92)
- Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditioner (NeurIPS 2025 · similarity 0.88)
- A New Perspective on Shampoo's Preconditioner (ICLR 2025 · similarity 0.88)
- Structured Preconditioners in Adaptive Optimization: A Unified Analysis (ICML 2025 · similarity 0.87)
- Error Feedback Can Accurately Compress Preconditioners (ICML 2024 · similarity 0.87)
- Universal Sequence Preconditioning (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
