Research map / Architectures and efficiency / RNNs and normalization
Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| average the independently refined worker weights (ablation) | SWAP · CIFAR10 | test accuracy (%): 94.70 ± 0.20 -> 95.23 ± 0.08 | Helped |
| average the independently refined worker weights (ablation) | SWAP · CIFAR100 | test accuracy (%): 75.74 ± 0.15 -> 78.18 ± 0.21 | Helped |
| average the independently refined worker weights (ablation) | SWAP · ImageNet | Top1 accuracy (%): 75.96 ± 0.02 -> 76.19 ± 0.03 | Helped |
| average the independently refined worker weights (ablation) | SWAP · ImageNet | Top5 accuracy (%): 93.15 ± 0.02 -> 93.32 ± 0.02 | Helped |
| use large-batch SGD instead of small-batch SGD | SGD · CIFAR10 | test accuracy (%): 95.24 ± 0.09 -> 94.77 ± 0.23 | Hurt |
| use large-batch SGD instead of small-batch SGD | SGD · CIFAR100 | test accuracy (%): 77.01 ± 0.25 -> 75.84 ± 0.35 | Hurt |
| use large-batch SGD instead of small-batch SGD | SGD · ImageNet | Top1 accuracy (%): 76.14± 0.07 -> 75.86 ± 0.03 | Hurt |
| use a large-batch phase followed by parallel small-batch refinement and weight averaging | SWAP · CIFAR10 | training time (sec): 254.12 ± 0.62 -> 169.20 ± 3.25 | Helped |
| use a large-batch phase followed by parallel small-batch refinement and weight averaging | SWAP · CIFAR100 | test accuracy (%): 77.01 ± 0.25 -> 78.18 ± 0.21 | Helped |
| use a large-batch phase followed by parallel small-batch refinement and weight averaging | SWAP · ImageNet | training time (min): 235.29 ± 0.33 -> 156.55 ± 0.56 | Helped |
| use large-batch SWA rather than SWAP's parallel refinement | CIFAR100 SWA · CIFAR100 | test accuracy (%): 76.06 ± 0.25 -> 76.00± 0.31 | No clear effect |
| refine large-batch training with sequential small-batch SWA rather than parallel SWAP | CIFAR100 · CIFAR100 | training time (sec): 125.30 ± 1.85 -> 398.0 ± 1.35 | Hurt |
Closest papers by meaning
- Trainable Weight Averaging: Efficient Training by Optimizing Historical Solutions (ICLR 2023 · similarity 0.91)
- Diverse Weight Averaging for Out-of-Distribution Generalization (NeurIPS 2022 · similarity 0.88)
- When, Where and Why to Average Weights? (ICML 2025 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.86)
- Model Merging by Uncertainty-Based Gradient Matching (ICLR 2024 · similarity 0.85)
- Why Do We Need Weight Decay in Modern Deep Learning? (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.