Research map / Optimization / Optimizers
SWALP : Stochastic Weight Averaging in Low Precision Training
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Chris De Sa
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Average low-precision SGD iterates with SWALP | 8-bit Small-block BFP training · CIFAR-10, VGG16 | test error (%): 7.61 $\pm{0.15}$ -> 6.70 $\pm{0.12}$ | Improved |
| Average low-precision SGD iterates with SWALP | 8-bit Small-block BFP training · CIFAR-10, PreResNet-164 | test error (%): 5.83 $\pm{0.05}$ -> 5.01 $\pm{0.14}$ | Improved |
| Average low-precision SGD iterates with SWALP | 8-bit Small-block BFP training · CIFAR-100, VGG16 | test error (%): 29.59 $\pm{0.32 }$ -> 26.65 $\pm{0.29}$ | Improved |
| Average low-precision SGD iterates with SWALP | 8-bit Small-block BFP training · CIFAR-100, PreResNet-164 | test error (%): 23.97 $\pm{0.08}$ -> 21.76 $\pm{0.28}$ | Improved |
| Average low-precision SGD iterates with SWALP | ResNet-18 low-precision training · ImageNet | Top-1 error (%): 36.56 -> 34.89 | Improved |
| Use Small-block rather than Big-block BFP (ablation) | SWALP · CIFAR-10 and CIFAR-100 | test error (%): CIFAR-10 VGG16: 7.36 $\pm{0.05}$ -> 6.70 $\pm{0.12}$; CIFAR-10 PreResNet-164: 5.61 $\pm{0.17}$ -> 5.01 $\pm{0.14}$; CIFA | Improved |
| Average every batch instead of once per epoch (ablation) | SWALP, VGG-16 · CIFAR-100 after one epoch | test error (%): 30.00% -> 28.85% | Improved |
| Increase averaging frequency (ablation) | SWALP, VGG-16 · CIFAR-100 after 20 epochs | test error (%) | No clear effect |
| Compute the averaged model in 9-bit rather than full precision (ablation) | SWALP, VGG-16 with 8-bit Small-block BFP · CIFAR-100 | test error (%) | No clear effect |
| Store and compute the averaged model in 8-bit precision (ablation) | SWALP, VGG-16 with 8-bit Small-block BFP · CIFAR-100 | test error (%): 26.85% vs full precision averaging | Hurt |
| Use averaging precision below 8 bits (ablation) | SWALP, VGG-16 with 8-bit Small-block BFP · CIFAR-100 | test error (%) | Hurt |
| Average low-precision SGD iterates for 30 rather than 10 epochs (ablation) | SWALP, ResNet-18 · ImageNet | Top-1 error (%): 34.89 -> 34.34 | Improved |
Closest papers by meaning
- Generalization Analysis of Stochastic Weight Averaging with General Sampling (ICML 2024 · similarity 0.86)
- Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.84)
- SGD with memory: fundamental properties and stochastic acceleration (ICLR 2025 · similarity 0.83)
- Scaling Laws for Floating–Point Quantization Training (ICML 2025 · similarity 0.83)
- Investigating the Role of Weight Decay in Enhancing Nonconvex SGD (CVPR 2025 · similarity 0.83)
- Collage: Light-Weight Low-Precision Strategy for LLM Training (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
