Research map / Optimization / Nonconvex and smooth optimization
Training Deep Networks without Learning Rates Through Coin Betting
Francesco Orabona, Tatiana Tommasi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COCOB-Backprop instead of tuned learning-rate optimizers | Deep-network training on MNIST, CIFAR-10, and PTB · MNIST, CIFAR-10, PTB | training cost | Mixed |
| Use COCOB-Backprop instead of tuned learning-rate optimizers | Two-layer fully connected network and CNN · MNIST | test error rate | Mixed |
| Use COCOB-Backprop instead of tuned learning-rate optimizers | TensorFlow CNN · CIFAR-10 | training cost | No clear effect |
| Use COCOB-Backprop instead of tuned learning-rate optimizers | TensorFlow CNN · CIFAR-10 | test error rate: ~0.008 in error rate | No clear effect |
| Use COCOB-Backprop instead of tuned learning-rate optimizers | Medium LSTM · PTB | training cost (per-word perplexity) | Mixed |
| Use COCOB-Backprop instead of tuned learning-rate optimizers | Medium LSTM · PTB | test cost (per-word perplexity) | Mixed |
| Remove the sigmoid transformation from the parameter update (ablation) | COCOB-Backprop · Experiments in the paper | empirical results | Improved |
| Bound the update denominator below by αL (ablation) | COCOB-Backprop · Deep-network training | initial parameter movement | Improved |
| Return the last iterate instead of an average or a random iterate (ablation) | COCOB-Backprop · Deep-network training | training or test performance | No clear effect |
Closest papers by meaning
- Better Parameter-Free Stochastic Optimization with ODE Updates for Coin-Betting (AAAI 2022 · similarity 0.88)
- Coin Betting and Parameter-Free Online Learning (NeurIPS 2016 · similarity 0.86)
- Deep Learning Games (NeurIPS 2016 · similarity 0.86)
- Neumann Optimizer: A Practical Optimization Algorithm for Deep Neural Networks (ICLR 2018 · similarity 0.86)
- Deep Learning with Logged Bandit Feedback (ICLR 2018 · similarity 0.86)
- Learning Gradient Descent: Better Generalization and Longer Horizons (ICML 2017 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
