Research map / Optimization / Optimizers
PASO: Step Parallel Stochastic Optimization
Jianrong Lu, Zhuoya Gu, Haobo Li, Zhiyu Zhu, Yechao Zhang, Jianhai Chen, Minghui Yang, Junwei Liu, Jian Wang, Qinming He, Hui LIU, Junhui Hou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PASO step-parallel solving to Adam/AdamW | GPT-2 on WikiText-2 · WikiText-2 (pre-trained) | iterations: 1000 -> 48 (20.8×) | Improved |
| add PASO to Adam | GPT-2 on WikiText-2 · WikiText-2 (pre-trained) | test perplexity: 20.4 -> 20.7 | No clear effect |
| add PASO to Adam | ResNet50 on CIFAR-10 · CIFAR-10 (from scratch) | top-1 accuracy: 90.1 -> 89.5 | Hurt |
| add PASO to AdamW | ViT on CIFAR-10 · CIFAR-10 (from scratch) | top-1 accuracy: 78.0 -> 78.4 | No clear effect |
| add PASO to Adam for diffusion training | UNet on LSUN Church · LSUN Church 128x128 | FID: 9.08 (600k iters) -> 9.55 (136k iters) | Mixed |
| increase window size p from 7 to 21 (ablation) | AdamW+PASO on GPT-2 · WikiText-2 | iterations: 143 -> 48 | Improved |
| increase window size p from 7 to 21 (ablation) | AdamW+PASO on GPT-2 · WikiText-2 | wall-clock speedup: 4.0× -> 4.4× | No clear effect |
| increase batch size from 52 to 112 (ablation) | AdamW+PASO on GPT-2 · WikiText-2 | wall-clock speedup: 3.2× -> 4.4× | Improved |
| vary tolerance delta in [1e-7, 0.1] and EMA decay in [0.9, 0.9999] (ablation) | PASO with Adam on CIFAR-10 · CIFAR-10 | iterations and quality | No clear effect |
| vary learning rate from 4e-4 to 1e-2 (ablation) | PASO on CNN CIFAR-10 · CIFAR-10 | accuracy | No clear effect |
| use PASO step parallelism instead of unoptimized data parallelism | GPT-2 Adam · WikiText-2 | wall-clock time: 381s -> 139s | Improved |
| use PASO prototype instead of optimized DDP/DeepSpeed | GPT-2 Adam · WikiText-2 | wall-clock time: 139s vs 140s (DDP), 138s (DeepSpeed) | No clear effect |
Closest papers by meaning
- SGD with Adaptive Preconditioning: Unified Analysis and Momentum Acceleration (ICLR 2026 · similarity 0.88)
- EIPO: Efficient Inter-Step Parallel Optimization (NeurIPS 2026 · similarity 0.88)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.86)
- Incremental Gradient Descent with Small Epoch Counts is Surprisingly Slow on Ill-Conditioned Problems (ICML 2025 · similarity 0.86)
- Coupling-based Convergence Diagnostic and Stepsize Scheme for Stochastic Gradient Descent (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
