Research map / Optimization / Optimizers
Neural Taylor Approximations: Convergence and Exploration in Rectifier Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use neural Taylor losses as convex snapshots of network optimization | Rectifier networks trained with an optimizer having O(√N) regret · Rectifier networks | regret per round: O(1/√N) | Helped |
| Use RMSProp instead of SGD | Rectifier networks · MNIST autoencoder | training loss | Helped |
| Use RMSProp instead of SGD | Rectifier networks · MNIST autoencoder | activation-configuration exploration | Helped |
| Use RMSProp instead of SGD | Rectifier networks · CIFAR-10 | training loss | No clear effect |
| Use Adam instead of RMSProp | Rectifier networks · MNIST autoencoder | training loss | Helped |
| Use Adam instead of RMSProp | Rectifier networks · MNIST autoencoder | activation-configuration exploration | Hurt |
| Use Adam, RMSProp, or SGD | Rectifier networks · MNIST and CIFAR-10 experiments | regret growth: pN | No clear effect |
Closest papers by meaning
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks (ICLR 2019 · similarity 0.86)
- The Hidden Convex Optimization Landscape of Regularized Two-Layer ReLU Networks: an Exact Characterization of Optimal Solutions (ICLR 2022 · similarity 0.86)
- On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widths (ICML 2021 · similarity 0.86)
- A Sufficient Condition for Convergences of Adam and RMSProp (CVPR 2019 · similarity 0.86)
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.