Research map / Learning theory and algorithms / Prediction and decision losses
Towards a Mathematical Understanding of the Difficulty in Learning With Feedforward Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the approximate Newton/GGN update instead of BP | MLP (2,10,10,4) · Four regions classification | convergence behavior: 61.1 sec. for 1000 BP iterations; 1314.1 sec. for 1000 AN/GGN iterations; 21.4 times | Mixed |
| Use SoftPlus activation (ablation) | AN/GGN on MLP (2,20,1) · Four regions classification | converged loss behavior | Helped |
| Use Bent identity activation (ablation) | AN/GGN on MLP (2,20,1) · Four regions classification | converged loss behavior | Helped |
| Use Sigmoid activation (ablation) | AN/GGN on MLP (2,20,1) · Four regions classification | converged loss behavior | Hurt |
| Use tanh activation (ablation) | AN/GGN on MLP (2,20,1) · Four regions classification | converged loss behavior | Hurt |
| Use SoftSign activation rather than the other tested squashing activations (ablation) | AN/GGN on MLP (2,20,1) · Four regions classification | converged loss behavior | Helped |
Closest papers by meaning
- Practical Gauss-Newton Optimisation for Deep Learning (ICML 2017 · similarity 0.90)
- Rethinking Gauss-Newton for learning over-parameterized models (NeurIPS 2023 · similarity 0.89)
- Theoretical Characterisation of the Gauss Newton Conditioning in Neural Networks (NeurIPS 2024 · similarity 0.88)
- Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.87)
- Convergence Analysis of Newton's Method for Neural Networks in the Overparameterized Limit (NeurIPS 2026 · similarity 0.87)
- Gradient Regularized Newton Boosting Trees with Global Convergence (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.