Research map / Optimization / Nonconvex and smooth optimization
On the Ineffectiveness of Variance Reduced Optimization for Deep Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SVRG instead of SGD | LeNet on CIFAR10 · CIFAR10 | test error | Helped |
| Apply SVRG instead of SGD | DenseNet on CIFAR10 and ResNet-18 on ImageNet · CIFAR10 and ImageNet | test error | Hurt |
| Use SCSG instead of SGD | CIFAR10 and ImageNet comparisons · CIFAR10 and ImageNet | test error | Hurt |
| Switch from SGD to SVRG for fine-tuning | ResNet-50 on ImageNet · ImageNet | test accuracy | No clear effect |
| Switch from SGD to SVRG for fine-tuning | DenseNet-169 on ImageNet · ImageNet | test accuracy | No clear effect |
| Reuse each data point's augmentation transform after the snapshot pass (ablation) | SVRG on LeNet · CIFAR10 | gradient variance | Helped |
| Use different augmentation transforms after the snapshot pass (ablation) | SVRG on LeNet · CIFAR10 | gradient variance | Hurt |
| Average the tail of each epoch for the next snapshot (ablation) | SVRG and SGD · CIFAR10 | test error | Hurt |
| Average the tail of each epoch for the next snapshot (ablation) | SVRG and SGD · CIFAR10 | initial convergence speed | Helped |
| Replace ReLU with ELU (ablation) | SVRG on DenseNet · CIFAR10 | gradient variance | No clear effect |
| Replace ReLU with ELU (ablation) | SVRG · CIFAR10 | gradient variance | Helped |
| Use a mega-batch snapshot with 10 inner steps (ablation) | Streaming SVRG on DenseNet · CIFAR10 | gradient variance | Hurt |
Closest papers by meaning
- A Coefficient Makes SVRG Effective (ICLR 2025 · similarity 0.93)
- Stochastic Variance Reduction for Nonconvex Optimization (ICML 2016 · similarity 0.90)
- Variance Reduction for Faster Non-Convex Optimization (ICML 2016 · similarity 0.90)
- A Simple Stochastic Variance Reduced Algorithm with Fast Convergence Rates (ICML 2018 · similarity 0.88)
- Hybrid Stochastic-Deterministic Minibatch Proximal Gradient: Less-Than-Single-Pass Optimization with Nearly Optimal Generalization (ICML 2020 · similarity 0.88)
- Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.