Research map / Optimization / Nonconvex and smooth optimization
Adaptive Normalized Risk-Averting Training for Deep Neural Networks
Zhiguang Wang, Tim Oates, James Lo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MSE/CE training with ANRAT | ConvNets · MNIST | test error %: 0.93 -> 0.52 | Improved |
| add dropout and ReLU to ANRAT ConvNets (ablation) | ConvNets + ANRAT · MNIST | test error %: 0.52 -> 0.39 | Improved |
| replace CE/MSE training with ANRAT | ConvNets + dropout · CIFAR-10 | accuracy %: 80.58 -> 85.15 | Improved |
| replace MSE with ANRAT | Deep MLP · MNIST | test error %: 1.91 -> 1.45 | Improved |
| replace CE with ANRAT | Deep MLP · MNIST | test error %: 2.4 -> 1.45 | Improved |
| replace MSE with ANRAT | Shallow MLP · MNIST | test error %: 2.02 -> 1.94 | No clear effect |
| replace CE with ANRAT | Shallow MLP · MNIST | test error %: 1.93 -> 1.94 | No clear effect |
| use ANRAT instead of unsupervised pretraining | Deep MLP · MNIST | test error %: 1.45 vs 1.41 and 1.2 | Hurt |
| replace MSE with ANRAT | Denoised auto-encoder · uniform vs Gaussian block masking noise | reconstruction error | Mixed |
Closest papers by meaning
- Layer-wise Conditioning Analysis in Exploring the Learning Dynamics of DNNs (ECCV 2020 · similarity 0.87)
- A General Regret Bound of Preconditioned Gradient Method for DNN Training (CVPR 2023 · similarity 0.87)
- Benefit of deep learning with non-convex noisy gradient descent: Provable excess risk bound and superiority to kernel methods (ICLR 2021 · similarity 0.87)
- Adversarial Margin Maximization Networks (TPAMI 2019 · similarity 0.86)
- Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training (NeurIPS 2021 · similarity 0.86)
- Global Optimality Beyond Two Layers: Training Deep ReLU Networks via Convex Programs (ICML 2021 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
