Research map / Architectures and efficiency / RNNs and normalization
Training independent subnetworks for robust prediction
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshminarayanan, Andrew Mingbo Dai, Dustin Tran
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MIMO with three subnetworks instead of a deterministic network | ResNet28-10 · CIFAR10 | NLL: 0.159 -> 0.123107 | Improved |
| Use MIMO with three subnetworks instead of a deterministic network | ResNet28-10 · CIFAR100 | accuracy: 79.8 -> 81.9819 | Improved |
| Use MIMO with two subnetworks and input repetition (ρ=0.6) instead of a deterministic network | ResNet50 · ImageNet | accuracy: 76.1 -> 77.5 | Improved |
| Use MIMO with two subnetworks and input repetition (ρ=0.6) instead of a deterministic network | ResNet50 · ImageNet-C | ECE: 0.105 -> 0.106 | Hurt |
| Increase the number of subnetworks; the ensemble improves before capacity limits individual members (ablation) | MIMO ResNet28-10 · CIFAR10 and CIFAR100 | ensemble performance | Mixed |
| Add correlated input repetition during training (ρ>0) (ablation) | MIMO ResNet50 · ImageNet | performance | Improved |
| Increase input repetition toward ρ=1 (ablation) | MIMO ResNet50 · ImageNet | ensemble benefit | Hurt |
| Increase minibatch repetition (ablation) | MIMO · CIFAR and ImageNet experiments | performance | Improved |
| Increase regularization strength (ablation) | MIMO ResNet28-10 · CIFAR10 and CIFAR100 | accuracy and log-likelihood | Hurt |
| Increase ensemble size under stronger regularization (ablation) | MIMO ResNet28-10 · CIFAR10 and CIFAR100 | accuracy and log-likelihood | Hurt |
| Use separate inputs for each output rather than shared input in a naive multihead network (ablation) | MIMO · CIFAR10 | subnetwork diversity | Improved |
Closest papers by meaning
- Robustness via Cross-Domain Ensembles (ICCV 2021 · similarity 0.87)
- Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity (ICLR 2022 · similarity 0.87)
- On the Certified Robustness for Ensemble Models and Beyond (ICLR 2022 · similarity 0.86)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.86)
- Improving Adversarial Robustness via Promoting Ensemble Diversity (ICML 2019 · similarity 0.86)
- Prune and Tune Ensembles: Low-Cost Ensemble Learning with Sparse Independent Subnetworks (AAAI 2022 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
