Research map / Optimization / Optimizers
HyperAdam: A Learnable Task-Adaptive Adam for Network Training
Shipeng Wang, Jian Sun, Zongben Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · MNIST basic MLP with ReLU | final loss: 0.32 -> 0.29 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · MNIST basic MLP with ELU | final loss: 0.31 -> 0.28 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · MNIST basic MLP with tanh | final loss: 0.33 -> 0.36 | Hurt |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · LSTM sequence prediction, baseline task | loss: 0.49 -> 0.42 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · LSTM sequence prediction, small noise | loss: 0.32 -> 0.19 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · LSTM sequence prediction, 2-layer | loss: 0.27 -> 0.26 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · CNN-1, MNIST, trained for 10000 steps | loss: 0.10 -> 0.05 | Improved |
| Use task-adaptive Adam updates and combine multiple candidates with learned weights | HyperAdam · CNN-2, MNIST, trained for 10000 steps | top-2: 99.80% -> 99.78% | Hurt |
| Add the StateCell LSTM block (ablation) | HyperAdam · 3-hidden-layer MLP on MNIST | training loss | Improved |
| Add gradient preprocessing in StateCell (ablation) | HyperAdam · 3-hidden-layer MLP on MNIST | training loss | Improved |
| Use both Random Scaling and Combination with Convex Functions (ablation) | HyperAdam · 3-hidden-layer MLP on MNIST | training loss | Improved |
| Increase candidate updates from one to multiple (J=5, 10, 15, 25, or 35) (ablation) | HyperAdam · 4-hidden-layer MLP on MNIST | cross-entropy loss | Improved |
Closest papers by meaning
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.88)
- MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence (NeurIPS 2024 · similarity 0.88)
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.86)
- SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training (ICLR 2025 · similarity 0.85)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.85)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
