Research map / Optimization / Implicit bias and generalization
A Method for Enhancing Generalization of Adam by Multiple Integrations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MIAdam, which adds multiple-integral (cumulative summation) terms to Adam's first moment for the early phase of training to escape sharp minima and find flatter minima, then switches back to Adam for convergence.
Problem
Adam often generalizes worse than SGD and its variants, and existing Adam variants either fail to retain Adam's fast convergence or generalize only modestly.
Approach
Treat the optimizer trajectory as a signal and apply multiple integrations (cumulative summations with rate kappa) to Adam's first-order momentum, which filters out sharp minima and biases training toward flat regions; switch to plain Adam after a threshold step zeta to ensure convergence.
Key results
- MIAdam1 reaches 94.20% on CIFAR-10 with ResNet18 vs 93.89% for Adam, and 75.03% vs 73.17% on CIFAR-100.
- On ImageNet-1k with ResNet18, MIAdam1 achieves 72.27% top-1 vs 67.19% for Adam.
- On CIFAR-10 with 80% label noise, MIAdam1 reaches 67.68% vs 66.39% for Adam; gains are larger at lower noise levels (e.g., 87.67% vs 84.90% at 40%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a first-order multiple integral term to Adam (MIAdam1) | Adam · CIFAR-10 and CIFAR-100 across ResNet18, ResNet50, DenseNet121, and PyramidNet110 | top-1 test accuracy: CIFAR-10: 93.89_{\pm 0.18} -> 94.20_{\pm 0.12} (ResNet18); CIFAR-100: 73.17_{\pm0.26} -> 75.03_{\pm0.05} (ResNet18) | Helped |
| Add a first-order multiple integral term to Adam (MIAdam1) | Adam · ImageNet-1k | top-1 test accuracy: 46.48 -> 52.34 (AlexNet); 67.19 -> 72.27 (ResNet18); 71.48 -> 75.39 (DenseNet121) | Helped |
| Add a first-order multiple integral term to Adam (MIAdam1) | Adam · R8, R52, and Movie Review with BERT and RoBERTa | test accuracy: 98.15_{ \pm 0.02} -> 98.18_{ \pm 0.03} (R8 BERT); 87.72_{ \pm 0.09} -> 89.73_{ \pm 0.17} (Movie Review RoBERTa) | Helped |
| Add a first-order multiple integral term to Adam (MIAdam1) | Adam · CIFAR-10 with 20% and 40% label noise | top-1 test accuracy: 88.24 -> 90.32 (20%); 84.90 -> 87.67 (40%) | Helped |
| Use second-order instead of first-order multiple integration (MIAdam2) (ablation) | MIAdam · CIFAR-10 and CIFAR-100 classification | top-1 test accuracy: ResNet50 CIFAR-10: 94.17_{\pm 0.10} -> 94.28_{\pm 0.15}; DenseNet121 CIFAR-100: 77.02_{\pm0.10} -> 76.21_{\pm0.33} | Mixed |
| Use third-order instead of first-order multiple integration (MIAdam3) (ablation) | MIAdam · CIFAR-10 and CIFAR-100 classification | top-1 test accuracy: ResNet18 CIFAR-100: 75.03_{\pm0.05} -> 74.53_{\pm0.22}; PyramidNet110 CIFAR-100: 72.51_{\pm0.24} -> 71.34_{\pm0.17} | Mixed |
| Use second- or third-order multiple integration instead of first-order integration (ablation) | MIAdam · ImageNet-1k | top-1 test accuracy: AlexNet: 52.34 -> 49.61 (MIAdam2) and 52.34 -> 43.36 (MIAdam3); ResNet18: 72.27 -> 70.70 (MIAdam2) and 72.27 -> 70.31 (M | Mixed |
| Use second- or third-order multiple integration instead of first-order integration (ablation) | MIAdam · CIFAR-10 under label noise | top-1 test accuracy: 90.32 -> 89.13 (20% noise) and 90.32 -> 88.71 (20% noise); 87.67 -> 85.03 (40% noise) and 87.67 -> 85.87 (40% noise) | Mixed |
| Use multiple integral updates throughout training without switching back to Adam (ablation) | MIAdam1 · Convex optimization convergence analysis | regret per step convergence criterion | Hurt |
| Switch from multiple-integral updates to Adam after the specified early training period (ablation) | MIAdam · CIFAR-100 with DenseNet121 | epochs to steady-state convergence | No clear effect |
Surprising findings
- Adding only a first-order integration (MIAdam1 without switching) makes the optimizer non-convergent while it hovers around flat minima, which motivates switching back to Adam.
- Higher integration order does not always help: MIAdam3 shows more oscillation near flat regions than MIAdam2 in simulations.
Limitations
- Convergence theorem shows the pre-switch multiple-integral phase has non-vanishing regret, so guarantees rely on switching to Adam after a threshold step.
- Label-noise experiments run once per noise level; text experiments use only three datasets; several hyperparameters (kappa, zeta) tuned by grid search.
- Diffusion-theory generalization result relies on quadratic-loss and low-temperature assumptions.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k R8 R52 MR
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k R8 R52 MR
Metrics: top-1 accuracy test accuracy training time Hessian eigenvalues escape time
Baselines: Adam NAdam AdamW ND-Adam Adamax AdaBound SWATS Adai
Builds on: Adam SWATS AdaBound Adai diffusion theory of escape from minima
Closest papers by meaning
- A Comprehensive Framework for Analyzing the Convergence of Adam: Bridging the Gap with SGD (ICML 2025 · similarity 0.89)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.87)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.87)
- MADA: Meta-Adaptive Optimizers Through Hyper-Gradient Descent (ICML 2024 · similarity 0.86)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.85)
- Adam with model exponential moving average is effective for nonconvex optimization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.