AI Paper Atlas

Research map / Optimization / Optimizers

HyperAdam: A Learnable Task-Adaptive Adam for Network Training

Shipeng Wang, Jian Sun, Zongben Xu

AAAI 2019 · Optimization › Optimizers · approach: Adam convergence and adaptive learning rates · 23 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · MNIST basic MLP with ReLUfinal loss: 0.32 -> 0.29Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · MNIST basic MLP with ELUfinal loss: 0.31 -> 0.28Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · MNIST basic MLP with tanhfinal loss: 0.33 -> 0.36Hurt
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · LSTM sequence prediction, baseline taskloss: 0.49 -> 0.42Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · LSTM sequence prediction, small noiseloss: 0.32 -> 0.19Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · LSTM sequence prediction, 2-layerloss: 0.27 -> 0.26Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · CNN-1, MNIST, trained for 10000 stepsloss: 0.10 -> 0.05Improved
Use task-adaptive Adam updates and combine multiple candidates with learned weightsHyperAdam · CNN-2, MNIST, trained for 10000 stepstop-2: 99.80% -> 99.78%Hurt
Add the StateCell LSTM block (ablation)HyperAdam · 3-hidden-layer MLP on MNISTtraining lossImproved
Add gradient preprocessing in StateCell (ablation)HyperAdam · 3-hidden-layer MLP on MNISTtraining lossImproved
Use both Random Scaling and Combination with Convex Functions (ablation)HyperAdam · 3-hidden-layer MLP on MNISTtraining lossImproved
Increase candidate updates from one to multiple (J=5, 10, 15, 25, or 35) (ablation)HyperAdam · 4-hidden-layer MLP on MNISTcross-entropy lossImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy