Research map / Architectures and efficiency / RNNs and normalization
Learning the Number of Neurons in Deep Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add group sparsity regularizer to learn neuron counts (ablation) | BNetC · ImageNet | Top-1 accuracy (%): 61.1 -> 62.7 | Helped |
| add group sparsity regularizer (ablation) | Dec8 · ImageNet | Top-1 accuracy (%): 64.8 -> 64.8 | No clear effect |
| start Dec8 with 640 neurons per layer and apply GS (ablation) | Dec8-640 · ImageNet | Top-1 accuracy (%): 66.9 -> 68.6 | Helped |
| start Dec8 with 768 neurons per layer and apply GS (ablation) | Dec8-768 · ImageNet | Top-1 accuracy (%): 68.1 -> 68.0 | Hurt |
| use sparse group Lasso instead of group sparsity (ablation) | Dec8-640 · ImageNet | Top-1 accuracy (%): 68.6 -> 67.5 | Hurt |
| overcomplete start (M=640 vs 512) before regularization (ablation) | Dec8 · ImageNet | Top-1 accuracy (%): 64.8 -> 68.6 | Helped |
| apply regularization: training-validation gap (ablation) | Dec8-768 · ImageNet | training-validation gap (%): 14% -> 10% | Helped |
| vary lambda_l in 0.051..0.51 (20 pairs) (ablation) | Dec8-768 GS · ImageNet | validation accuracy: std of 0.33% | No clear effect |
| apply GS to Dec3 with 1D filters vs 2D kernels | Dec3 · ICDAR | Top-1 accuracy: 83.8% -> 89.3% | Helped |
| apply GS regularization (ablation) | Dec3 · ICDAR | Top-1 accuracy: 89.3% -> 90.1% | Helped |
| apply SGL regularization (ablation) | Dec3 · ICDAR | Top-1 accuracy: 89.3% -> 89.9% | Helped |
| remove zeroed neurons at test time | Ours-Dec3-GS · batch size 16 runtime | relative speed-up (%): 49.63 | Helped |
Closest papers by meaning
- Redundant representations help generalization in wide neural networks (NeurIPS 2022 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.85)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.84)
- On the Nonlinearity of Layer Normalization (ICML 2024 · similarity 0.84)
- Expand-and-Cluster: Parameter Recovery of Neural Networks (ICML 2024 · similarity 0.84)
- On the VC dimension of deep group convolutional neural networks (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.