Research map / Optimization / Nonconvex and smooth optimization
Scaled Conjugate Gradient Method for Nonconvex Optimization in Deep Neural Networks
Naoki Sato, Koshiro Izumi, Hideaki Iiduka
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use constant learning rates instead of diminishing learning rates (ablation) | SCG methods · CIFAR-100/CIFAR-10 ResNet-18 and IMDb LSTM | training loss and accuracy | Improved |
| use SCGAdam instead of other optimizers with constant learning rate | ResNet-18 · CIFAR-100 | training loss | Improved |
| use SCGAdam instead of other optimizers with constant learning rate | ResNet-18 · CIFAR-10 | training loss | Improved |
| use Momentum instead of SCGAdam | ResNet-18 · CIFAR-100 (200 epochs) | test accuracy | Hurt |
| use SCGAdam instead of other optimizers | ResNet-18 · CIFAR-10 | epochs to reach 90% test accuracy: 90 % | Improved |
| use SCGAdam or AMSGrad instead of other optimizers | ResNet-18 · CIFAR-100 | epochs to reach 70% test accuracy: 70 % | Improved |
| use SCGAdam alongside RMSprop | LSTM · IMDb | training loss | Improved |
| use SCGAMSGrad instead of Adam-type baselines | DCGAN · LSUN-Bedroom | FID | Improved |
| use SCGAMSGrad instead of other adaptive optimizers | SNGAN · CIFAR10 | FID | Improved |
| use SCGAdam instead of other adaptive optimizers | WGAN-GP · Tiny ImageNet | FID | Improved |
Closest papers by meaning
- Adaptive Methods for Nonconvex Optimization (NeurIPS 2018 · similarity 0.90)
- Towards Better Understanding of Adaptive Gradient Algorithms in Generative Adversarial Nets (ICLR 2020 · similarity 0.88)
- Two-Timescale Gradient Descent Ascent Algorithms for Nonconvex Minimax Optimization (JMLR 2025 · similarity 0.86)
- Nonconvex Stochastic Bregman Proximal Gradient Method with Application to Deep Learning (JMLR 2025 · similarity 0.86)
- Non-asymptotic Analysis of Biased Adaptive Stochastic Approximation (NeurIPS 2024 · similarity 0.85)
- ASGO: Adaptive Structured Gradient Optimization (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
