Research map / Optimization / Implicit bias and generalization
Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Geometric Parameterization (GmP), which writes ReLU neuron weights in hyperspherical coordinates (radius and angles) to stabilize characteristic activation boundaries under stochastic optimization, and reports gains on UCI, ImageNet32/VGG-6 and ImageNet/ResNet-18.
Problem
Characteristic activation boundaries of ReLU neurons are unstable under stochastic gradient noise in common Cartesian parameterizations and normalizations (SP, WN, BN, LN), which slows convergence and hurts generalization.
Approach
Reparameterize each ReLU unit as r·ReLU(u(θ)^T x + λ) with a unit direction u(θ) in hyperspherical coordinates, so the CAB's angular change is bounded by the angular perturbation norm (≤ ||ε||), and add a parameter-free input mean normalization (IMN) for intermediate layers.
Key results
- On 7 UCI regression benchmarks, GmP achieves the lowest test RMSE on most datasets (e.g., Energy 0.474±0.013 vs 0.898 for SP).
- On ImageNet32 with VGG-6 at batch size 1024, GmP+IMN reaches 42.70% top-1 vs 41.39% for BN; at batch size 512 it reaches 43.62% top-1.
- On ImageNet with ResNet-18, GmP+IMN achieves 67.24% top-1 and 87.19% top-5 versus 66.85% and 86.92% for BN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GmP instead of the strongest listed baseline | MLP, 100 hidden units · UCI Boston, Concrete, Energy, Power, Wine, and Yacht | test RMSE: 3.370\pm0.145 -> 3.057\pm0.144; 5.472\pm0.144 -> 5.153\pm0.098; 0.898\pm0.274 -> 0.474\pm0.013; 4.065\pm0.029 -> 4.022\p | Helped |
| Use GmP instead of BN | MLP, 100 hidden units · UCI Naval | test RMSE: 0.001\pm0.000 -> 0.003\pm0.001 | Hurt |
| Add IMN to GmP (ablation) | VGG-6 · ImageNet32, batch size 1024 | top-1 validation accuracy (%): 41.29\pm0.08 -> 42.70\pm0.15 | Helped |
| Add IMN to GmP (ablation) | VGG-6 · ImageNet32, batch size 1024 | top-5 validation accuracy (%): 65.49\pm0.06 -> 66.98\pm0.18 | Helped |
| Use GmP+IMN instead of BN | VGG-6 · ImageNet32, batch size 1024 | top-1 validation accuracy (%): 41.39\pm0.02 -> 42.70\pm0.15 | Helped |
| Use GmP+IMN instead of BN | VGG-6 · ImageNet32, batch size 1024 | top-5 validation accuracy (%): 65.89\pm0.06 -> 66.98\pm0.18 | Helped |
| Use GmP+IMN instead of BN | ResNet-18 · ImageNet, single-center-crop validation | top-1 validation accuracy (%): 66.85\pm0.05 -> 67.24\pm0.21 | Helped |
| Use GmP+IMN instead of BN | ResNet-18 · ImageNet, single-center-crop validation | top-5 validation accuracy (%): 86.92\pm0.02 -> 87.19\pm0.15 | Helped |
Surprising findings
- A small non-residual VGG-6 with GmP+IMN (43.62% top-1) is reported to outperform WRN 28-2 trained with BN (43.08%) on ImageNet32.
- GmP's stability allows a 10x larger learning rate, while the gains over BN on ImageNet are modest (about 0.4 points top-1).
Limitations
- Analysis restricted to single-hidden-layer ReLU networks; multi-layer extension is left for future work.
- Experiments are limited to small CNN/MLP architectures and do not cover transformers or NLP.
- Gains over BN on ImageNet/ResNet-18 are modest, and the paper notes the potential for sparse activations or vanishing gradients from the product structure.
Experimental setup
Datasets: UCI benchmarks (Boston, Concrete, Energy, Naval, Power, Wine, Yacht) 1D Levy regression 2D Banana classification ImageNet32 ImageNet (ILSVRC 2012)
Benchmarks: UCI regression ImageNet32 ImageNet (ILSVRC 2012) 1D Levy regression 2D Banana classification
Metrics: test RMSE top-1 validation accuracy top-5 validation accuracy test accuracy convergence speed
Baselines: Standard Parameterization Weight Normalization Batch Normalization WN+MBN Layer Normalization
Builds on: Weight Normalization Batch Normalization Adam Mean-only Batch Normalization
Closest papers by meaning
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.90)
- Activation-Descent Regularization for Input Optimization of ReLU Networks (ICML 2024 · similarity 0.89)
- Topological obstruction to the training of shallow ReLU neural networks (NeurIPS 2024 · similarity 0.88)
- Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon (NeurIPS 2025 · similarity 0.88)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)
- Most ReLU Networks Admit Identifiable Parameters (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.