Research map / Optimization / Optimizers
HVAdam: A Full-Dimension Adaptive Optimizer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HVAdam, an adaptive optimizer that tracks a 'hidden vector' capturing the stable gradient trend (via gradient projection with restarts) and uses it both as an update direction and to precondition per-dimension learning rates, aiming to fix the valley dilemma of Adam-type methods.
Problem
Adaptive optimizers such as Adam adjust learning rates per coordinate and so ignore the global loss-landscape structure, leading to slow convergence in non-axis-aligned valley-like regions (the "valley dilemma").
Approach
Maintain a hidden vector v_t, updated by projecting onto the momentum direction, to approximate the invariant gradient trend; add a restart rule based on a moving average of cosine similarity between v_t and m_t; and build a hidden-vector-based preconditioner from (g_t - v_{t-1})^2 to set per-dimension step sizes, with an extra step along v_t.
Key results
- On ResNet50 ImageNet, HVAdam reaches 77.22% top-1, second only to Lookaround (77.32%) and above SGDM (76.49%) and Adam (72.87%).
- On ViT-B/16, HVAdam achieves 99.00% on CIFAR-10 and 92.38% on CIFAR-100, the best among compared optimizers (Lookaround 98.71 / 92.21).
- On SN-GAN with ResNet generator on CIFAR-10, HVAdam achieves the lowest FID of 12.72 ± 0.21 versus 12.98 for AdaBelief and 13.01 for Adam.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hidden vector to Adam (ablation) | Adam optimizer on 1-layer LSTM · Penn TreeBank valid | perplexity: 85.04 -> 84.54 | Helped |
| add hidden-vector-based learning-rate adjustment (ablation) | HVAdam with hidden vector on 1-layer LSTM · Penn TreeBank valid | perplexity: 84.54 -> 83.42 | Helped |
| add restart strategy (ablation) | HVAdam^2 on 1-layer LSTM · Penn TreeBank valid | perplexity: 83.42 -> 83.31 | Helped |
| use HVAdam instead of Lookaround | ViT-B/16 · CIFAR10 | test accuracy: 98.71 -> 99.00 | Helped |
| use HVAdam instead of Lookaround | ViT-B/16 · CIFAR100 | test accuracy: 92.21 -> 92.38 | Helped |
| use HVAdam instead of Lookaround | ResNet50 · ImageNet val | Top-1 accuracy: 77.32 -> 77.22 | Hurt |
| use HVAdam instead of AdaBelief | SN-GAN with ResNet generator · CIFAR-10 | FID: 12.98$\pm$0.22 -> 12.72$\pm$0.21 | Helped |
Surprising findings
- HVAdam's FID with WGAN is reported as better than other optimizers' FID with WGAN-GP.
Limitations
- The algorithm is more complex and incurs extra memory overhead, which may limit use in memory-constrained settings.
- Many hyperparameters (e.g., the 0.1 restart threshold and the lr mapping) were chosen empirically.
- ResNet50 ImageNet result was not extensively hyperparameter tuned due to computational cost.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet Penn TreeBank IWSLT14
Benchmarks: CIFAR-10 CIFAR-100 ImageNet Penn TreeBank IWSLT14 FID on CIFAR-10
Metrics: top-1 accuracy perplexity FID regret bound
Baselines: SGD Adam AdaBelief AdamW RAdam RMSProp Yogi AdaBound Fromage MSVAG SWA Lookahead
Builds on: Adam AdaBelief RMSProp Lookaround Adai
Closest papers by meaning
- AdaptiveMix: Improving GAN Training via Feature Space Shrinkage (CVPR 2023 · similarity 0.86)
- Enhancing Optimizer Stability: Momentum Adaptation of The NGN Step-size (NeurIPS 2025 · similarity 0.85)
- E$^2$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation (ICML 2024 · similarity 0.85)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.84)
- MCGAN: Enhancing GAN Training with Regression-Based Generator Loss (AAAI 2025 · similarity 0.84)
- Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.