Research map / Optimization / Optimizers
Win: Weight-Decay-Integrated Nesterov Acceleration for Faster Network Training
Pan Zhou, Xingyu Xie, Zhouchen Lin, Kim-Chuan Toh, Shuicheng Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Win acceleration | AdamW · ImageNet ResNet18 | top-1 accuracy (%): 67.9 -> 71.0 | Improved |
| add Win2 (three-step) acceleration | LAMB · ImageNet ResNet18 | top-1 accuracy (%): 68.5 -> 71.3 | Improved |
| add Win acceleration | Adam · ImageNet ResNet18 | top-1 accuracy (%): 66.5 -> 69.3 | Improved |
| add Win acceleration | LAMB · ImageNet ResNet50 (avg epochs) | top-1 accuracy (%): 78.7 -> 79.4 | Improved |
| add Win acceleration | AdamW · ImageNet ViT-B (avg epochs) | top-1 accuracy (%): 80.7 -> 81.7 | Improved |
| add Win acceleration | AdamW · COCO Mask R-CNN Swin | box AP average: 51.5 -> 51.7 | Improved |
| add Win acceleration | AdamW · Penn Treebank LSTM | test perplexity: 67.0 -> 66.5 | Improved |
| add Win2 acceleration | Adam · WikiText-103 Transformer-XL | test perplexity: 26.7 -> 25.0 | Improved |
| replace Nesterov acceleration with Win (ablation) | AdamW · ImageNet ResNet18 | top-1 accuracy (%): 69.3 -> 71.0 | Improved |
| replace Nesterov acceleration with Win (ablation) | AdamW · ImageNet ResNet50 (100 epochs) | top-1 accuracy (%): 78.4 -> 78.0 | Hurt |
| increase update steps from q=3 to q=6 (ablation) | AdamW-Win2 · ImageNet ResNet18 | top-1 accuracy (%): 71.2 -> 71.0 | No clear effect |
| increase update steps from q=3 to q=6 (ablation) | AdamW-Win2 · ResNet18 peak GPU memory (M) | peak memory: 8094 -> 8588 | Hurt |
Closest papers by meaning
- EMA-Nesterov: Stabilizing Nesterov's Lookahead for Accelerated Deep Learning Optimization (NeurIPS 2026 · similarity 0.88)
- Accelerating neural network training: An analysis of the AlgoPerf competition (ICLR 2025 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.86)
- Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.86)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- Implicit Bias of AdamW: $\ell_\infty$-Norm Constrained Optimization (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
