Research map / Optimization / Optimizers
Small Steps and Giant Leaps: Minimal Newton Solvers for Deep Learning
Joao F. Henriques, Sebastien Ehrhardt, Samuel Albanie, Andrea Vedaldi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CurveBall instead of SGD | VGG-f · ImageNet-100 | best validation error (%): 39.8 -> 33.8 | Improved |
| use CurveBall instead of Adam | VGG-f · ImageNet-100 | best validation error (%): 35.9 -> 33.8 | Improved |
| use CurveBall instead of SGD | ResNet-18 · CIFAR-10 | best validation error (%): 12.8 -> 16.1 | Hurt |
| add dropout 0.3 to CurveBall lambda | ResNet-18 · CIFAR-10 | validation error (%): 15.3 -> 13.5 | Improved |
| add automatic lambda rescaling (ablation) | CurveBall · CIFAR-10 basic CNN | training error (%): 15.3 -> 14.1 | Improved |
| add automatic lambda rescaling (ablation) | CurveBall · CIFAR-10 basic CNN + BN | training error (%): 9.4 -> 7.6 | Improved |
| add automatic lambda rescaling (ablation) | CurveBall · CIFAR-10 ResNet-18 | training error (%): 1.3 -> 0.7 | Improved |
| add automatic lambda rescaling (ablation) | CurveBall · ImageNet-100 VGG-f | training error (%): 12.7 -> 10.3 | Improved |
| add automatic lambda rescaling (ablation) | CurveBall · CIFAR-10 basic CNN + BN | validation error (%): 15.8 -> 16.3 | Hurt |
| add batch normalisation | CurveBall · CIFAR-10 basic CNN | best validation error (%): 19.3 -> 15.8 | Improved |
| use CurveBall instead of SGD | Basic CNN · CIFAR-10 | training error (%): 18.9 -> 15.3 | Improved |
| use CurveBall instead of Adam on the noisy Rosenbrock U[0,3] | CurveBall · Stochastic Rosenbrock U[0,3] | iterations to solution: 2750 -> 13 | Improved |
Closest papers by meaning
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.89)
- Second-Order Neural ODE Optimizer (NeurIPS 2021 · similarity 0.88)
- Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness (ICLR 2024 · similarity 0.88)
- Skip the Hessian, Keep the Rates: Globalized Semismooth Newton with Lazy Hessian Updates (NeurIPS 2026 · similarity 0.87)
- Sassha: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation (ICML 2025 · similarity 0.87)
- Gradient descent with generalized Newton’s method (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
