Research map / Optimization / Optimizers
MetaOptimize: A Framework for Optimizing Step Sizes and Other Meta-parameters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaOptimize, a causal (backward-view, eligibility-trace style) framework that tunes step sizes and other differentiable meta-parameters online by minimizing a discounted sum of future losses, wrapping any first-order base optimizer.
Problem
Hyperparameter (step-size) tuning is expensive via grid search, and fixed or hand-designed learning rate schedules are suboptimal, especially in continual/non-stationary learning where step sizes must track changing optima.
Approach
Define a discounted sum of future losses as the objective and derive a causal backward-view gradient estimate via eligibility-trace-style recursion (matrix H_t), then feed it into any first-order meta-update (SGD, Adam, Lion); Hessian-free and 2x2/L approximations make it cheap. Special cases include IDBD and hypergradient descent (gamma=0).
Key results
- On CIFAR10 with ResNet-18, every tested Hessian-free MetaOptimize combination outperformed its well-tuned fixed-step-size baseline, and it was robust to initial step sizes orders of magnitude smaller than optimal.
- On non-stationary CIFAR100 (10 tasks), MetaOptimize with scalar or blockwise step sizes consistently outperformed AdamW with best fixed step size in cumulative accuracy.
- On ImageNet, MetaOptimize (AdamW, Lion) added about 44% time and 33% memory overhead over AdamW, and it outperformed DoG, gdtuo, Prodigy and mechanic; on TinyStories it beat all baselines except a well-tuned cosine schedule within 30k iterations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt step sizes online with MetaOptimize instead of using fixed step sizes | ResNet-18 · CIFAR10 | performance | Helped |
| use blockwise step sizes instead of a scalar step size (ablation) | MetaOptimize · CIFAR10 | performance | Mixed |
| adapt step sizes online with MetaOptimize instead of using the best fixed step size | AdamW · Non-stationary CIFAR100 | cumulative top-1 accuracy | Helped |
| use blockwise step sizes instead of a scalar step size (ablation) | MetaOptimize · Non-stationary CIFAR100 | cumulative top-1 accuracy | Helped |
| use blockwise step sizes instead of scalar step sizes (ablation) | MetaOptimize · ImageNet | performance | No clear effect |
| use MetaOptimize step-size adaptation instead of the compared baselines | MetaOptimize · ImageNet | performance | Helped |
| use MetaOptimize step-size adaptation instead of the compared baselines | MetaOptimize · TinyStories | performance | Helped |
| replace the full trace calculation with the 2×2 approximation (ablation) | MetaOptimize · Experiments (reported generally) | performance | No clear effect |
| replace the 2×2 approximation with the L-approximation (ablation) | MetaOptimize · Experiments (reported generally) | performance | No clear effect |
| use a smaller discount factor γ (ablation) | MetaOptimize · Stationary supervised learning | performance | Hurt |
| set γ to 0.999 instead of 1 (ablation) | MetaOptimize · Non-stationary CIFAR100 | performance | Helped |
Surprising findings
- Performance was robust to initial step sizes that were several orders of magnitude smaller than the optimal fixed step size.
- Blockwise step sizes gave no improvement over scalar step sizes on ImageNet, though they helped on CIFAR10.
Limitations
- Experiments restricted to Hessian-free approximations; Hessian-based variants unexplored.
- Backward-view approximation may degrade for large meta step sizes; theory for gamma=1 requires workaround.
- Only differentiable meta-parameters (step sizes) studied; discrete meta-parameters not covered.
Experimental setup
Datasets: CIFAR10 CIFAR100 ImageNet TinyStories
Benchmarks: CIFAR10 Non-stationary CIFAR100 ImageNet TinyStories
Metrics: test accuracy cumulative top-1 accuracy loss learning curves per-iteration wall-clock time overhead GPU memory overhead
Baselines: AdamW with tuned fixed step size Lion with tuned fixed step size AdamW with cosine schedule DoG gdtuo Prodigy mechanic Hypergradient descent
Builds on: IDBD Hypergradient descent Delta-bar-Delta Lion Adam
Closest papers by meaning
- Meta-Learning with Adaptive Hyperparameters (NeurIPS 2020 · similarity 0.89)
- Guarantees for Tuning the Step Size using a Learning-to-Learn Approach (ICML 2021 · similarity 0.89)
- $\mu$LO: Compute-Efficient Meta-Generalization of Learned Optimizers (ICLR 2026 · similarity 0.89)
- Memory-Reduced Meta-Learning with Guaranteed Convergence (AAAI 2025 · similarity 0.88)
- Delve into the Applicability of Advanced Optimizers for Multi-Task Learning (NeurIPS 2026 · similarity 0.86)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.