Research map / Optimization / Optimizers
Improving Deep Learning Speed and Performance Through Synaptic Neural Balance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes synaptic neural balancing, a scaling operation equalizing input and output regularizer costs per neuron, and shows full or partial balancing improves speed and accuracy of FCNs and RNNs, especially with limited data.
Problem
Training neural networks faster and more accurately, especially with limited data, via a regularization approach based on rebalancing weights across neurons without changing network function.
Approach
For BiLU/homogeneous activations, rescaling a neuron's incoming weights by λ and outgoing by 1/λ preserves function; choosing λ* = (||OUT||_p/||IN||_p)^(1/2p) equalizes in/out L_p costs. Iterated balancing converges to a unique fixed point (strictly convex problem in log-scale variables), and can be interleaved with SGD as a regularizer.
Key results
- On MNIST with 2-layer FCN, L1 partial balancing reaches 94.54% test accuracy vs 91.22% plain.
- With full balancing before training, 5-layer FCN L1 reaches 95.5% vs 89.1% without balancing (Table 1).
- On 1% MNIST data, L1 neural balancing gives up to 10-fold faster convergence and 3-10% higher accuracy than plain models (Discussion).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add full balance before training (ablation) | FCNs trained on MNIST · MNIST test | accuracy: 90.1% -> 91.2% | Helped |
| add full balance before training (ablation) | FCNs trained on MNIST with L1 regularization · MNIST test | accuracy: 90.1% -> 94% | Helped |
| add full balance before training (ablation) | FCNs trained on MNIST with L2 regularization · MNIST test | accuracy: 90.1% -> 91.2% | Helped |
| use partial L1 neural balancing instead of plain training | 2-layer FCN · MNIST test | accuracy: 91.22% -> 94.542% | Helped |
| use partial L1 neural balancing instead of plain training | 3-layer FCN · MNIST test | accuracy: 90.84% -> 93.94% | Helped |
| use partial L1 neural balancing instead of plain training | 5-layer FCN · MNIST test | accuracy: 91.37% -> 96.26% | Helped |
| use partial L2 neural balancing instead of plain training | 2-layer FCN · MNIST test | accuracy: 91.22% -> 91.19% | No clear effect |
| use partial L2 neural balancing instead of plain training | 5-layer FCN · MNIST test | accuracy: 91.37% -> 91.63% | Helped |
| add full balance before training (ablation) | 3-layer RNN trained on IMDB · IMDB test | accuracy: 88.26 -> 88.64% | Helped |
| add full balance before training (ablation) | 3-layer RNN trained with L1 regularization · IMDB test | accuracy: 88.23 -> 88.24% | No clear effect |
| add full balance before training (ablation) | 3-layer RNN trained with L2 regularization · IMDB test | accuracy: 88.22 -> 88.57% | Helped |
| use partial L1 neural balancing instead of plain training | 2-layer FCN · MNIST with 1% training data | accuracy: 84.15% -> 91.25% | Helped |
Surprising findings
- L1 regularization with 1% data collapses to 11.35% accuracy for 5-layer FCN, while L1 neural balancing stays around 90%.
Limitations
- Applies only to neurons with homogeneous (BiLU-like) activations, not directly to convolution layers.
- Experiments limited to small FCNs and a 3-layer RNN on MNIST, FashionMNIST, and IMDB.
Experimental setup
Datasets: MNIST FashionMNIST IMDB circle toy dataset
Benchmarks: MNIST FashionMNIST IMDB
Metrics: test accuracy convergence speed (epochs)
Baselines: Plain training L1 regularization L2 regularization L2 neural balancing
Builds on: Du, Hu, and Lee 2018 balance preservation Yang et al. 2022 neural balance-like regularization Stock et al. 2022 local heterosynaptic learning L1/L2 regularization
Closest papers by meaning
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks (ICML 2024 · similarity 0.87)
- Structured flexibility in recurrent neural networks via neuromodulation (NeurIPS 2024 · similarity 0.86)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.86)
- The impact of allocation strategies in subset learning on the expressive power of neural networks (ICLR 2025 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.86)
- Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.