Research map / Architectures and efficiency / RNNs and normalization
Expected Gradients of Maxout Networks and Consequences to Parameter Initialization
Hanna Tseran, Guido Montufar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use maxout initialization (c = 0.55555) instead of ReLU He initialization | Fully-connected maxout network trained with SGD · MNIST test | accuracy: 53.22^{\pm 24.08} -> 97.8^{\pm 0.15} | Improved |
| Use maxout initialization instead of ReLU He initialization | Fully-connected maxout network trained with SGD · Iris test | accuracy: 91.67^{\pm 3.73} -> 91.67^{\pm 3.73} | No clear effect |
| Use maxout initialization instead of ReLU He initialization | Convolutional maxout network trained with SGD · MNIST test | accuracy: 99.49^{\pm 0.04} -> 99.59^{\pm 0.04} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with SGD · CIFAR-10 test | accuracy: 91.7^{\pm 0.17} -> 91.21^{\pm 0.13} | Hurt |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with SGD · CIFAR-100 test | accuracy: 65.33^{\pm 0.27} -> 65.39^{\pm 0.39} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with SGD · Fashion MNIST test | accuracy: 93.55^{\pm 0.13} -> 93.49^{\pm 0.13} | No clear effect |
| Use maxout initialization instead of ReLU He initialization | Convolutional maxout network trained with SGD · SVHN test | accuracy: 96.74^{\pm 0.03} -> 97.78^{\pm 0.02} | Improved |
| Use maxout initialization instead of naive initialization | Fully-connected maxout network trained with Adam · MNIST test, all epochs | accuracy: 98.13^{\pm 0.09} -> 98.12^{\pm 0.10} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with Adam · MNIST test, all epochs | accuracy: 99.53^{\pm 0.04} -> 99.47^{\pm 0.07} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with Adam · Fashion MNIST test, all epochs | accuracy: 93.57^{\pm 0.17} -> 93.45^{\pm 0.10} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with Adam · CIFAR-10 test, all epochs | accuracy: 91.33^{\pm 0.31} -> 91.06^{\pm 0.22} | No clear effect |
| Use maxout initialization instead of max-pooling initialization | Convolutional maxout network trained with Adam · CIFAR-100 test, all epochs | accuracy: 65.33^{\pm 1.26} -> 61.96^{\pm 0.58} | Hurt |
Closest papers by meaning
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.86)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.86)
- Scaling Exponents Across Parameterizations and Optimizers (ICML 2024 · similarity 0.85)
- On the Parameterization of Second-Order Optimization Effective towards the Infinite Width (ICLR 2024 · similarity 0.85)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
