Research map / Optimization / Nonconvex and smooth optimization
A Coefficient Makes SVRG Effective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SVRG's variance reduction term should be scaled down (and decayed over training) for deep networks, and proposes α-SVRG, which applies a linearly decaying coefficient to SVRG with AdamW or SGD to lower training loss across vision architectures and datasets.
Problem
SVRG is theoretically appealing but has shown limited practical benefit for training deep neural networks; with coefficient fixed at 1, its variance reduction term can increase gradient variance in later training and hurt convergence.
Approach
Derive the control-variate optimal coefficient for SVRG's variance reduction term, observe it is smaller for deeper models and decreases during training, and approximate it with a preset linearly decaying scalar coefficient α (α-SVRG) that is cheap to compute.
Key results
- On ImageNet-1K, α-SVRG lowers training loss relative to AdamW for all six models (e.g., ConvNeXt-F 3.487 to 3.467), whereas standard SVRG raises it for most models (e.g., ViT-B 2.817 to 3.309).
- On eight small datasets with ConvNeXt-F, α-SVRG lowers training loss on all datasets (e.g., Pets 2.20 to 1.96) and raises validation accuracy (e.g., Pets 72.8 to 77.8), while SVRG degrades Pets accuracy to 17.6.
- With three seeds, α-SVRG's mean training loss is lower than AdamW on all reported datasets, usually by more than one standard deviation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace AdamW with α-SVRG using linear coefficient decay | ConvNeXt-F · ImageNet-1K | training loss: 3.487 -> 3.467 | Helped |
| Replace AdamW with α-SVRG using linear coefficient decay | ImageNet-1K models · ImageNet-1K | training loss: 3.443 -> 3.415; 3.427 -> 3.392; 3.149 -> 3.097; 2.817 -> 2.806; 2.644 -> 2.642 | Helped |
| Replace AdamW with α-SVRG using linear coefficient decay | ConvNeXt-F · ImageNet-1K | validation accuracy: 76.0 -> 76.3; 73.9 -> 74.2; 74.3 -> 74.8; 76.4 -> 76.1; 81.6 -> 81.6; 83.7 -> 83.1 | Mixed |
| Replace AdamW with α-SVRG using linear coefficient decay | ConvNeXt-F · CIFAR-100, Pets, Flowers, STL-10, Food-101, DTD, SVHN, EuroSAT | training loss: 2.66 -> 2.62; 2.20 -> 1.96; 2.40 -> 2.16; 1.64 -> 1.57; 2.45 -> 2.42; 1.98 -> 1.83; 1.59 -> 1.57; 1.25 -> 1.23 | Helped |
| Replace AdamW with α-SVRG using linear coefficient decay | ConvNeXt-F · CIFAR-100, Pets, Flowers, STL-10, Food-101, DTD, SVHN, EuroSAT | validation accuracy: 81.0 -> 81.4; 72.8 -> 77.8; 80.8 -> 83.3; 82.3 -> 84.0; 85.9 -> 85.9; 57.9 -> 61.8; 94.9 -> 95.8; 98.1 -> 98.2 | Helped |
| Use standard SVRG with coefficient 1 instead of AdamW | ImageNet-1K models · ImageNet-1K | training loss: 3.487 -> 3.505; 3.443 -> 3.431; 3.427 -> 3.389; 3.149 -> 3.172; 2.817 -> 3.309; 2.644 -> 3.113 | Mixed |
| Use standard SVRG with coefficient 1 instead of AdamW | ConvNeXt-F · CIFAR-100, Pets, Flowers, STL-10, Food-101, DTD, SVHN, EuroSAT | training loss: 2.66 -> 2.94; 2.20 -> 3.42; 2.40 -> 2.26; 1.64 -> 1.90; 2.45 -> 3.03; 1.98 -> 2.01; 1.59 -> 1.64; 1.25 -> 1.25 | Mixed |
| Vary initial coefficient α₀ across the reported values (ablation) | α-SVRG with ConvNeXt-F · small image classification datasets | training loss | Mixed |
| Use double schedules rather than global schedules (ablation) | α-SVRG with ConvNeXt-F · STL-10 | training loss: 1.64; 1.59; 1.61; 1.62; 2.07; 1.97; 1.81; 1.64 | Hurt |
| Reduce batch size (ablation) | α-SVRG with ConvNeXt-F · STL-10 | training loss | Mixed |
| Vary inner loop size between snapshot captures (ablation) | α-SVRG with ConvNeXt-F · STL-10 | training loss | Mixed |
| Apply α-SVRG only during the first 10% of training, then disable it (ablation) | ConvNeXt-F · small image classification datasets | training loss: 2.659 -> 2.644; 2.203 -> 2.190; 2.400 -> 2.328; 1.641 -> 1.616; 2.451 -> 2.444; 1.980 -> 1.918; 1.588 -> 1.583; 1.247 -> | Helped |
Surprising findings
- Standard SVRG increases training loss on most ImageNet-1K models and drops Pets validation accuracy to 17.6%, while the same SVRG framework with a decaying coefficient consistently helps.
- Applying the coefficient within each epoch (double schedules) gives a higher training loss than the AdamW baseline, while global schedules do not.
Limitations
- Computing the exact optimal coefficient is infeasible for large networks, so α-SVRG relies on a heuristic linear schedule.
- α-SVRG is ineffective with small batch sizes, requiring a sufficiently large batch size.
- Lower training loss does not always translate to better validation accuracy for larger models, which need extra regularization tuning.
Experimental setup
Datasets: ImageNet-1K CIFAR-10 CIFAR-100 Pets Flowers STL-10 Food-101 DTD SVHN EuroSAT
Benchmarks: ImageNet-1K CIFAR-100 Pets Flowers STL-10 Food-101 DTD SVHN EuroSAT
Metrics: final epoch training loss top-1 validation accuracy gradient variance
Baselines: AdamW SVRG SGD MARS
Builds on: SVRG AdamW ConvNeXt control variates MARS
Closest papers by meaning
- On the Ineffectiveness of Variance Reduced Optimization for Deep Learning (NeurIPS 2019 · similarity 0.93)
- SVRG and Beyond via Posterior Correction (ICML 2026 · similarity 0.90)
- Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimization (NeurIPS 2018 · similarity 0.89)
- Towards closing the gap between the theory and practice of SVRG (NeurIPS 2019 · similarity 0.89)
- Stochastic Variance Reduction for Nonconvex Optimization (ICML 2016 · similarity 0.88)
- MARS: Unleashing the Power of Variance Reduction for Training Large Models (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.