Research map / Optimization / Optimizers
On the Role of Batch Size in Stochastic Conditional Gradient Methods
Rustem Islamov, Roman Machacek, Aurelien Lucchi, Tony Silveti-Falls, Eduard Gorbunov, Volkan Cevher
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase batch size from 256 to 4096 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, seq len 1024 | final validation loss: 3.3717 -> 3.6472 | Hurt |
| increase batch size from 256 to 512 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.3717 -> 3.3765 | No clear effect |
| decrease batch size from 256 to 64 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.3717 -> 3.4554 | Hurt |
| increase batch size from 256 to 1024 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.3717 -> 3.4065 | Hurt |
| increase batch size from 256 to 2048 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.3717 -> 3.4799 | Hurt |
| reduce train sequence length from 1024 to 256 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, B=256 | final validation loss: 3.3717 -> 4.0441 | Hurt |
| reduce train sequence length from 1024 to 512 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, B=256 | final validation loss: 3.3717 -> 3.4796 | Hurt |
| increase train sequence length from 1024 to 2048 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, B=256 | final validation loss: 3.3717 -> 3.3792 | No clear effect |
| increase train sequence length from 1024 to 4096 at beta=3.6e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, B=256 | final validation loss: 3.3717 -> 3.4216 | Hurt |
| increase train sequence length from 1024 to 2048 at beta=1.2e-4 (ablation) | Scion 124M NanoGPT · 1.3B tokens, B=256 | final validation loss: 3.4587 -> 3.4126 | Improved |
| raise Frank-Wolfe stepsize from 1.2e-4 to 2.4e-4 at B=256 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.3857 -> 3.3706 | Improved |
| raise Frank-Wolfe stepsize from 1.2e-4 to 2.4e-4 at B=4096 (ablation) | Scion 124M NanoGPT · 1.3B tokens | final validation loss: 3.7715 -> 3.6694 | Improved |
Closest papers by meaning
- A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance (ICML 2021 · similarity 0.90)
- Optimal Rates for Multi-pass Stochastic Gradient Methods (JMLR 2017 · similarity 0.90)
- Stochastic Conditional Gradient Methods: From Convex Minimization to Submodular Maximization (JMLR 2020 · similarity 0.89)
- Trajectory of Mini-Batch Momentum: Batch Size Saturation and Convergence in High Dimensions (NeurIPS 2022 · similarity 0.89)
- Perfect Parallelization in Mini-Batch SGD with Classical Momentum Acceleration (NeurIPS 2026 · similarity 0.89)
- Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model (NeurIPS 2019 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
