Research map / Optimization / Optimizers
Regularized Adaptive Momentum Dual Averaging with an Efficient Inexact Subproblem Solver for Training Structured Neural Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAMDA, a regularized adaptive momentum dual averaging method with an implementable inexactness condition and a proximal-gradient subproblem solver, guaranteeing convergence and finite-time identification of the locally optimal group-sparse structure for training structured neural networks.
Problem
Training structured (e.g., group-sparse) neural networks with nonsmooth regularizers and adaptive preconditioners: the update subproblem has no closed form, and existing regularized adaptive methods do not guarantee that the output attains the locally optimal structure.
Approach
Combines momentum dual averaging (for variance reduction of gradients) with a dual-averaging-style cube-root accumulated-squared-gradient preconditioner, solves the regularized subproblem inexactly with proximal gradient under a checkable subgradient-norm criterion, and uses manifold identification theory to prove finite-time structure identification.
Key results
- On ImageNet/ResNet50, RAMDA reaches 74.53% accuracy with 29.19% weighted group sparsity, versus RMDA 74.47%/25.20% and ProxGen 74.17%/20.29%.
- On Transformer-XL/WikiText-103, RAMDA achieves perplexity 26.97 with 36.2% sparsity, versus RMDA 27.10 / 36.0% and ProxSGD 27.42 / 33.1%.
- The proposed PG subproblem solver is about 3x faster than ProxSSI with similar model quality on CIFAR experiments.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use RAMDA instead of RMDA | ResNet50 · ImageNet validation | accuracy: 74.47 $\pm$ 0.08\% -> 74.53 $\pm$ 0.10\% | Helped |
| use RAMDA instead of RMDA | ResNet50 · ImageNet | weighted group sparsity: 25.20 $\pm$ 1.69\% -> 29.19 $\pm$ 0.94\% | Helped |
| use RAMDA instead of RMDA | Transformer-XL (base) · WikiText-103 validation | perplexity: 27.10 $\pm$ 0.08 -> 26.97 $\pm$ 0.10 | Helped |
| use RAMDA instead of RMDA | Transformer-XL (base) · WikiText-103 | weighted group sparsity: 36.0 $\pm$ 2.7\% -> 36.2 $\pm$ 0.3\% | No clear effect |
| use RAMDA instead of ProxGen | Tacotron2 · LJSpeech validation | loss: 0.45 $\pm$ 0.01 -> 0.44 $\pm$ 0.01 | Helped |
| use RAMDA instead of RMDA | Tacotron2 · LJSpeech | weighted group sparsity: 45.9 $\pm$ 1.7\% -> 52.9 $\pm$ 1.6\% | Helped |
| enable early stopping for the PG subproblem solver (ablation) | RAMDA with VGG19 · CIFAR10 validation | accuracy: 92.7 $\pm$ 0.2\% -> 92.9 $\pm$ 0.2\% | No clear effect |
| enable early stopping for the PG subproblem solver (ablation) | RAMDA with VGG19 · CIFAR10 | weighted group sparsity: 86.7 $\pm$ 0.3\% -> 86.3 $\pm$ 0.4\% | Hurt |
| enable early stopping for the PG subproblem solver (ablation) | RAMDA with ResNet50 · CIFAR100 validation | accuracy: 69.9 $\pm$ 1.5\% -> 71.2 $\pm$ 1.4\% | Helped |
| enable early stopping for the PG subproblem solver (ablation) | RAMDA with ResNet50 · CIFAR100 | weighted group sparsity: 69.5 $\pm$ 2.1\% -> 67.5 $\pm$ 1.6\% | Hurt |
| replace Newton-Raphson subproblem solving with the PG solver | ProxGen on CIFAR10 · VGG19 / CIFAR10 | time/epoch: 79s -> 24s | Helped |
| replace Newton-Raphson subproblem solving with the PG solver | ProxGen on CIFAR100 · VGG19 / CIFAR100 | time/epoch: 79s -> 26s | Helped |
Surprising findings
- Early stopping of the proximal gradient subproblem solver does not noticeably hurt final accuracy or sparsity.
Limitations
- Convergence of iterates (beyond subsequence/point convergence) is left as future work since KL-based arguments do not apply to this method.
- Structure identification holds only asymptotically/locally and relies on assumptions such as prox-regularity and nondegeneracy.
- RAMDA's runtime is somewhat higher than non-adaptive baselines on some tasks (e.g., Transformer-XL 6954s vs 6167s per epoch).
Experimental setup
Datasets: ILSVRC 2012 ImageNet WikiText-103 LJSpeech CIFAR10 CIFAR100 MNIST
Benchmarks: ImageNet WikiText-103 LJSpeech CIFAR10 CIFAR100
Metrics: top-1 validation accuracy validation perplexity validation loss weighted group sparsity time per epoch
Baselines: RMDA ProxSGD ProxGen ProxSSI MSGD Adam
Builds on: RMDA MADGRAD ProxGen ProxSSI Adam
Closest papers by meaning
- Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · similarity 0.86)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.85)
- Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum (NeurIPS 2025 · similarity 0.85)
- Structured Inverse-Free Natural Gradient Descent: Memory-Efficient & Numerically-Stable KFAC (ICML 2024 · similarity 0.85)
- Memory-Efficient LLM Training with Online Subspace Descent (NeurIPS 2024 · similarity 0.85)
- Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.