Research map / Architectures and efficiency / Neural architecture search
Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
Shinichi Shirakawa, Yasushi Iwata, Youhei Akimoto
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the same mini-batch to compare sampled layer structures rather than different mini-batches (ablation) | AdaptiveLayer · MNIST | test error (%): 15.69 (29.9) -> 2.200 (0.125) | Improved |
| Initialize layer-skip probabilities at 0.5 instead of 0.969 (ablation) | AdaptiveLayer (b) · MNIST | test error (%): 35.26 (41.6) -> 15.69 (29.9) | Improved |
| Optimize layer-skip probabilities instead of keeping them fixed at 0.5 (ablation) | StochasticLayer · MNIST | test error (%): 4.704 (0.752) -> 2.200 (0.125) | Improved |
| Use deterministic rather than stochastic prediction (ablation) | AdaptiveLayer (a) · MNIST | test error (%): 2.200 (0.125) -> 2.218 (0.135) | No clear effect |
| Use deterministic rather than stochastic prediction (ablation) | AdaptiveConnection · CIFAR-100 | test error (%): 25.461 (0.408) -> 25.315 (0.409) | No clear effect |
| Optimize activation probabilities instead of keeping them fixed at 0.5 (ablation) | AdaptiveActivation · MNIST | test error (%): 1.452 (0.025) -> 1.407 (0.036) | Improved |
| Learn a mixture of ReLU and tanh activations instead of using ReLU for every unit (ablation) | AdaptiveActivation · MNIST | test error (%): 1.609 (0.044) -> 1.414 (0.054) | Improved |
| Learn a mixture of ReLU and tanh activations instead of using tanh for every unit (ablation) | AdaptiveActivation · MNIST | test error (%): 1.592 (0.069) -> 1.414 (0.054) | Improved |
| Use deterministic rather than stochastic prediction (ablation) | AdaptiveActivation · MNIST | test error (%): 1.414 (0.054) -> 1.407 (0.036) | No clear effect |
| Optimize network structures jointly during training rather than search them with Bayesian optimization at budget 10 | AdaptiveNet · MNIST | time (hour): 9.59 -> 1.01 | Improved |
| Optimize network structures jointly during training rather than use Bayesian optimization at budget 20 | AdaptiveNet · MNIST | test error (%): 1.490 -> 1.645 (0.072) | Hurt |
| Select DenseNet connections instead of keeping all connections (ablation) | AdaptiveConnection · CIFAR-10 | test error (%): 5.050 (0.147) -> 5.427 (0.167) | Hurt |
Closest papers by meaning
- Deep Neural Network Approximated Dynamic Programming for Combinatorial Optimization (AAAI 2020 · similarity 0.85)
- Constructing Deep Neural Networks by Bayesian Network Structure Learning (NeurIPS 2018 · similarity 0.84)
- The Dynamics of Learning: A Random Matrix Approach (ICML 2018 · similarity 0.84)
- On the Hardness of Training Deep Neural Networks Discretely (AAAI 2025 · similarity 0.83)
- On the Interplay between Graph Structure and Learning Algorithms in Graph Neural Networks (ICML 2025 · similarity 0.83)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
