Research map / Optimization / Implicit bias and generalization
Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK
Hongru Yang, Ziyu Jiang, Ruizhe Zhang, Yingbin Liang, Zhangyang Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize the bias to -0.5 instead of 0 (ablation) | 6-layer MLP (width 1024) · MNIST | accuracy (%): 97.9 -> 97.7 | No clear effect |
| initialize the bias to -1.0 instead of 0 (ablation) | 6-layer MLP (width 1024) · MNIST | accuracy (%): 97.9 -> 97.3 | Hurt |
| increase the magnitude of negative bias initialization (ablation) | 6-layer MLP (width 1024) · MNIST | activation sparsity level: bias 0 and -0.5: sparsity change within 2.5%; bias -1.0: within 10% | Improved |
| add Batch Normalization after ReLU in the sparsity-inducing MLP (ablation) | 6-layer MLP with large negative bias · MNIST | trainability | Improved |
| vary bias initialization (0, -0.1, -0.2, -0.3, -0.4) (ablation) | width-128 one-hidden-layer network · Toy linear regression, x in R^5 | training loss convergence rate | No clear effect |
Closest papers by meaning
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.89)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.88)
- Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods (ICLR 2025 · similarity 0.88)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.88)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
