Research map / Optimization / Implicit bias and generalization
Do Deep Neural Network Solutions Form a Star Domain?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Conjectures that SGD solution sets of neural networks form a star domain modulo permutations, and proposes Starlight to find a star model linearly connected to many independent solutions, with applications to BMA and model fusion.
Problem
Linear mode connectivity of SGD solutions (convexity conjecture) fails for narrower and deeper networks, requiring very wide nets; the paper asks whether a weaker star-shaped connectivity holds modulo permutations.
Approach
Introduces the star domain conjecture: there exists a star model linearly connected (modulo permutations) to all other SGD solutions, and the Starlight algorithm optimizes a model to have low expected loss on Monte-Carlo sampled linear paths to permuted source models, using weight matching at each epoch.
Key results
- Star-regular training-loss barrier on CIFAR-10 ResNet-18 is 0.078 versus 0.383 for regular-regular pairs.
- Star model trained on 50 CIFAR-100 source models reaches 78.4% test accuracy vs 77.3% for single regular models (ensemble 81.3%).
- BMA over the star domain gives better AUROC than deep ensemble baselines but worse ECE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a Starlight star model instead of comparing two independently trained models | ResNet-18 and other architectures · CIFAR-10, CIFAR-100, and ImageNet-1k | loss barrier: 0.383 ± 0.056 -> 0.078 ± 0.007 (CIFAR-10 ResNet-18); 5.948 -> 2.794 (ImageNet-1k ResNet-18) | Helped |
| Use an Adam-trained star model rather than an Adam-trained regular model pair | ResNet-18 · CIFAR-10 | loss barrier: 1.368 ± 0.551 -> 0.335 ± 0.022 | Helped |
| Increase the number of source models used to construct the star model | Starlight · Held-out models; source-set sizes 2 to 50 | loss barrier | Helped |
| Increase ResNet width | ResNet and WideResNet star-regular comparisons · CIFAR; widths 1× to 8× | loss barrier: roughly 0.004 compared to 0.012 at width 8x | Helped |
| Use Beta (2, 2) sampling instead of uniform interpolation sampling (ablation) | Starlight · CIFAR-10, ResNet-18 | star-regular loss barrier: 0.084 ± 0.012 -> 0.069 ± 0.010 | No clear effect |
| Use constant t = 0.5 instead of uniform interpolation sampling (ablation) | Starlight · CIFAR-10, ResNet-18 | star training loss: 0.001 ± 0.000 -> 0.082 ± 0.004 | Hurt |
| Increase interpolation points from 11 to 51 (ablation) | Loss-barrier evaluation · CIFAR-10, ResNet-18 | regular-regular loss barrier: 0.376 ± 0.055 -> 0.369 ± 0.053 | No clear effect |
| Increase interpolation points from 11 to 51 (ablation) | Loss-barrier evaluation · CIFAR-10, ResNet-18 | star-regular loss barrier: 0.090 ± 0.007 -> 0.094 ± 0.008 | No clear effect |
| Increase held-out sample size from 3 to 15 (ablation) | Starlight star-regular evaluation · CIFAR-10, ResNet-18 | star-regular average loss barrier: 0.090 ± 0.007 -> 0.093 ± 0.014 | No clear effect |
| Use an SGD-induced star model when connecting to Adam solutions (ablation) | Starlight star model · Adam-trained solutions | loss barrier | Helped |
| Use Sinkhorn-rebasin instead of weight matching (ablation) | VGG19 permutation matching · CIFAR-10 | loss barrier | Hurt |
| Use a Starlight star-domain posterior rather than deep-ensemble estimates | ResNet-18 Bayesian model averaging · CIFAR-10 | AUROC | Helped |
Surprising findings
- Star model barriers to held-out models decrease as the number of source models grows, without saturating at 50.
- Star-regular barriers remain about one third of regular-regular barriers across widths and depths.
Limitations
- Star domain conjecture is empirical and not theoretically proven; star-regular barriers remain non-zero.
- Star models are less accurate than full ensembles over the source set.
- Adam-trained solutions show weaker connectivity to the star model than SGD solutions, so the conjecture is scoped to SGD solutions.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 ImageNet-1k
Metrics: loss barrier test accuracy AUROC ECE
Baselines: Regular-regular barrier Deep ensemble Sinkhorn re-basin Best of n single model
Builds on: linear mode connectivity weight matching (re-basin) mode connectivity Sinkhorn re-basin parameter-curve fitting
Closest papers by meaning
- SGD Converges to Global Minimum in Deep Learning via Star-convex Path (ICLR 2019 · similarity 0.86)
- Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup (NeurIPS 2019 · similarity 0.85)
- STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.84)
- Can Neural Networks Achieve Optimal Computational-statistical Tradeoff? An Analysis on Single-Index Model (ICLR 2025 · similarity 0.83)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.83)
- Implicit bias of SGD in $L_2$-regularized linear DNNs: One-way jumps from high to low rank (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.