Research map / Optimization / Optimizers
MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MADA, which parameterizes a convex combination of adaptive optimizers (Adam, AMSGrad, Adan, Yogi, Lion) and learns the interpolation coefficients online via hyper-gradient descent, plus AVGrad (AMSGrad with averaging instead of max).
Problem
Adaptive optimizers each excel on different tasks and none uniformly beats Adam; offline optimizer search is costly and does not adapt during training.
Approach
Unify several adaptive optimizers into one parameterized update rule whose coefficients are interpolation weights in a unit hypercube, and learn those coefficients during training with hyper-gradient descent; replace AMSGrad's max with time-averaging (AVGrad) so hyper-gradients flow better.
Key results
- On OpenWebText with GPT-2 (125M), MADA-FS outperforms Adam with established parameters by 0.019 in validation loss.
- MADA beats Adam and HyperAdam by 4.45 and 2.37 validation perplexity points on Wikitext.
- HyperAdam performs almost identically to Adam (2.8950 vs. 2.8956 validation loss), suggesting gains come from interpolation coefficient adaptation rather than beta tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt optimizer interpolation coefficients during training | Adam, AVGrad, Yogi, Adan, and Lion parameterized optimizer (MADA) · OpenWebText | validation loss: +0.34 perplexity over Adam | Helped |
| adapt optimizer interpolation coefficients during training | MADA · Wikitext | perplexity: +4.45 perplexity over Adam; +2.37 over HyperAdam | Helped |
| adapt optimizer interpolation coefficients during training | MADA · Lambada | perplexity: 0.88 | Mixed |
| use the learned final optimizer statically from the beginning (MADA-FS) (ablation) | MADA · OpenWebText | validation loss: 0.004 | Helped |
| use the learned final optimizer statically from the beginning (MADA-FS) (ablation) | MADA · Shakespeare | training loss | Hurt |
| replace AMSGrad's maximum operator with time-averaging (AVGrad) (ablation) | MADA · validation | validation loss | Helped |
| tune only Adam's beta parameters with hyper-gradients (HyperAdam), rather than adapting optimizer interpolation coefficients (ablation) | Adam · OpenWebText | performance: 2.8950 vs. 2.8956 | No clear effect |
| initialize beta parameters suboptimally at beta1=0.7 and beta2=0.8 (ablation) | MADA · OpenWebText | validation loss | Helped |
| initialize beta parameters suboptimally at beta1=0.7 and beta2=0.8 (ablation) | Adam · OpenWebText | validation loss | Hurt |
| initialize MADA with beta3=0.9 instead of beta3=0 (ablation) | MADA · OpenWebText | beta1 coefficient: 0.99 | Mixed |
| interpolate between Adam and AVGrad rather than using either base optimizer alone | interpolated optimizer · theoretical convergence analysis | convergence bound | Helped |
Surprising findings
- Including AMSGrad among base optimizers hurts MADA's end performance, attributed to the max operator blocking hyper-gradient flow to beta2.
- HyperAdam (tuning only beta1, beta2 via hyper-gradients) gives essentially no gain over Adam.
Limitations
- Additional memory from maintaining a larger optimizer state per parameter.
- Theory covers only interpolation between AVGrad and Adam without momentum in full; full MADA lacks general convergence guarantees.
- Hyper-learning rates require tuning, though reportedly insensitive.
Experimental setup
Datasets: OpenWebText Shakespeare CIFAR-10 Wikitext Lambada
Benchmarks: OpenWebText validation loss Wikitext perplexity Lambada perplexity Shakespeare training loss CIFAR-10 test accuracy
Metrics: validation loss perplexity training loss test accuracy
Baselines: Adam HyperAdam Adan Lion AVGrad SGD with momentum
Builds on: Adam AMSGrad Adan Yogi Lion hyper-gradient descent (Baydin et al.) AdamW
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.87)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.87)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.87)
- Prodigy: An Expeditiously Adaptive Parameter-Free Learner (ICML 2024 · similarity 0.87)
- AdaFisher: Adaptive Second Order Optimization via Fisher Information (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.