Research map / Optimization / Implicit bias and generalization
On ADMM in Deep Learning: Convergence and Saturation-Avoidance
Jinshan Zeng, Shao-Bo Lin, Yuan Yao, Ding-Xuan Zhou
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ADMM instead of SGD to train sigmoid nets | SGD (sigmoid) · Approximation of x^2 | approximation error: 2.46e-4 -> 2.53e-9 | Improved |
| use ADMM instead of SGD to train sigmoid nets | SGD (sigmoid) · Approximation of product gate uv | approximation error: 1.13e-3 -> 2.62e-9 | Improved |
| use ADMM instead of SGD to train sigmoid nets | SGD (sigmoid) · Learning L1 radial function (noise 0.1) | test error: 4.58e-5 -> 1.69e-5 | Improved |
| use ADMM instead of SGD to train sigmoid nets | SGD (sigmoid) · Learning L2 radial function (noise 0.1) | test error: 9.33e-5 -> 9.28e-6 | Improved |
| use sigmoid-ADMM instead of ReLU Adam | Adam (ReLU) · Approximation of x^2 | approximation error: 3.33e-8 -> 2.53e-9 | Improved |
| use sigmoid-ADMM instead of ReLU SGDM | SGDM (ReLU) · Approximation of product gate uv | approximation error: 3.37e-7 -> 2.62e-9 | Improved |
| use sigmoid-ADMM instead of ReLU Adam | Adam (ReLU) · Learning L1 radial function (noise 0.1) | test error: 2.16e-5 -> 1.69e-5 | Improved |
| use sigmoid-ADMM instead of ReLU Adam | Adam (ReLU) · Learning L2 radial function (noise 0.1) | test error: 1.02e-5 -> 9.28e-6 | Improved |
| use sigmoid-ADMM instead of ReLU Adam (run time) | Adam (ReLU) · Approximation of x^2 | run time (s): 38.26 -> 9.47 | Improved |
| use sigmoid-ADMM instead of ReLU SGD (run time) | SGD (ReLU) · Learning L2 radial function (noise 0.1) | run time (s): 104.52 -> 47.36 | Improved |
| vary ADMM penalty beta, choosing beta=1 (ablation) | ADMM (sigmoid) · Approximation of x^2 | approximation error | Improved |
| vary initialization scheme (MSRA, orthogonal, LeCun, Xavier) (ablation) | ADMM (sigmoid) · Approximation of product gate uv | approximation error | No clear effect |
Closest papers by meaning
- Federated ADMM from Bayesian Duality (ICLR 2026 · similarity 0.88)
- A Convergence Theory for Deep Learning via Over-Parameterization (ICML 2019 · similarity 0.87)
- A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers (ECCV 2018 · similarity 0.86)
- Fast and Provable ADMM for Learning with Generative Priors (NeurIPS 2019 · similarity 0.86)
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out With ADMM (AAAI 2018 · similarity 0.86)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
