Research map / Architectures and efficiency / RNNs and normalization
Learning Neural Network Subspaces
Mitchell Wortsman, Maxwell C Horton, Carlos Guestrin, Ali Farhadi, Mohammad Rastegari
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train a line subspace with cosine-similarity regularization (β > 0) | Neural network lines · CIFAR-10, ImageNet | accuracy | Improved |
| Use a layerwise sampling variant (ablation) | Neural network subspaces · Image classification benchmarks | midpoint accuracy | Improved |
| Use a layerwise sampling variant (ablation) | Neural network subspaces · Image classification benchmarks | ensemble accuracy | Hurt |
| Increase the number of simplex endpoints (ablation) | Simplex midpoint · CIFAR-10, Tiny-ImageNet | accuracy | Improved |
| Use the simplex midpoint instead of a standard-trained model | Simplex · CIFAR-10 with label noise | accuracy | Improved |
| Use the simplex midpoint instead of a standard-trained model | Simplex · CIFAR-10 | expected calibration error | Improved |
| Train with multiple independent subspace samples per batch instead of one (ablation) | Lines and curves · Image classification benchmarks | accuracy | Improved |
| Add feature-similarity regularization between sampled subspace models (ablation) | Lines and curves · Image classification benchmarks | accuracy | Improved |
| Initialize line endpoints from the same shared weights instead of independent random initializations (ablation) | Learned line · CIFAR-10, ImageNet | endpoint distance | No clear effect |
| Add cosine-similarity diversity regularization to SWA checkpoints (ablation) | SWA · Image classification benchmarks | performance | No clear effect |
| Use a simplex midpoint instead of SWA | Layerwise simplex · Image classification benchmarks | accuracy | Mixed |
| Use two models from opposing ends of a learned line instead of two independently trained models | Learned line · Image classification benchmarks | ensemble accuracy | No clear effect |
Closest papers by meaning
- Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors (ICML 2020 · similarity 0.88)
- Does SGD really happen in tiny subspaces? (ICLR 2025 · similarity 0.87)
- Neural Networks Perform Sufficient Dimension Reduction (AAAI 2025 · similarity 0.86)
- Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning (NeurIPS 2024 · similarity 0.85)
- Training Bayesian Neural Networks with Sparse Subspace Variational Inference (ICLR 2024 · similarity 0.85)
- Lean and Mean Adaptive Optimization via Subset-Norm and Subspace-Momentum with Convergence Guarantees (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
