Research map / Optimization / Implicit bias and generalization
On the Stability and Scalability of Node Perturbation Learning
Naoki Hiratani, Yash Mehta, Timothy P Lillicrap, Peter E. Latham
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of output units Ly in node perturbation (NP) versus SGD | NP, linear regression · linear regression | training time to target error | Hurt |
| increase hidden layer size Lh for NP in one-hidden-layer linear networks | NP · one-hidden-layer linear network | minimum training time | No clear effect |
| increase output layer size Ly for NP in one-hidden-layer linear network | NP · one-hidden-layer linear network | minimum training time | Hurt |
| raise learning rate above critical value eta* | NP with teacher noise · one-hidden-layer linear network | iterations until error starts to increase: eta^-1.5 -> eta^-3 scaling | Hurt |
| add weight normalization (constant incoming weight norm per neuron) (ablation) | NP · one-hidden-layer linear network with teacher noise | convergence stability | Mixed |
| increase hidden layer width for NP vs SGD weight update alignment | NP · SARCOS | cosine similarity with SGD in hidden layer: falls off as 1/sqrt(Lh) | Hurt |
| increase learning rate for NP | NP · SARCOS | test error stability | Hurt |
| use SGD instead of NP regarding late instability | SGD · SARCOS | late-training instability | Improved |
Closest papers by meaning
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Improving Deep Learning Speed and Performance Through Synaptic Neural Balance (AAAI 2025 · similarity 0.87)
- Unconditional stability of a recurrent neural circuit implementing divisive normalization (NeurIPS 2024 · similarity 0.87)
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
