Research map / Optimization / Implicit bias and generalization
Input Space Mode Connectivity in Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends loss-landscape mode connectivity from parameter space to the input space of image classifiers, shows real/synthetic/adversarial inputs are connected via low-loss paths, and uses barrier statistics for adversarial detection.
Problem
Whether mode connectivity, known in parameter space, also holds in the input space of fixed deep networks, and whether this geometry can explain adversarial examples and aid detection and interpretability.
Approach
Treats inputs with low loss for a class as input-space modes, bypasses loss barriers between them by optimizing an intermediate point constrained to a hyperplane (fixed model, variable data), and uses barrier height on linear paths to separate real-real from real-adversarial pairs; conjectures geometric connectivity via high-dimensional percolation.
Key results
- On CIFAR-10 with VGG-16, the mode-connectivity detector reaches 91.4/96.9 (acc/AUC) on DeepFool and 93.7/98.3 on C&W, outperforming baselines on these advanced attacks but underperforming on FGSM (69.8/76.8).
- On ImageNet, real-adversarial pairs have mean max barrier 5.82 versus 2.27 for real-real pairs (5,000 pairs each).
- Synthetic and untrained-model (ResNet18 on CIFAR-10) inputs are connected by piecewise-linear low-loss paths below threshold 0.001 after two rounds of barrier optimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mode-connectivity features instead of the benchmark detectors | VGG-16 adversarial detection · CIFAR-10, Deepfool and C&W | accuracy/AUC (%) | Helped |
| Use mode-connectivity features instead of the benchmark detectors | VGG-16 adversarial detection · CIFAR-10, FGSM, BIM, and PGD | accuracy/AUC (%) | Hurt |
| Use mode-connectivity features instead of the benchmark detectors | VGG-16 adversarial detection · CIFAR-100, C&W | accuracy/AUC (%) | Helped |
| Use mode-connectivity features instead of the benchmark detectors | VGG-16 adversarial detection · CIFAR-100, FGSM, BIM, PGD, and Deepfool | accuracy/AUC (%) | Mixed |
| Remove logits from the mode-connectivity feature vectors (ablation) | Mode connectivity adversarial detector · CIFAR-10/100, FGSM, BIM, and PGD | accuracy/AUC (%) | Hurt |
| Remove logits from the mode-connectivity feature vectors (ablation) | Mode connectivity adversarial detector · CIFAR-10/100, Deepfool and C&W | accuracy/AUC (%) | No clear effect |
Surprising findings
- Real-adversarial pairs show much higher and more complex loss barriers than real-real pairs, enabling detection.
- Input-space connectivity appears even in randomly initialized networks, which the authors attribute to high-dimensional percolation.
Limitations
- Percolation-based conjecture is a proof sketch assuming independence between cube labels, which does not hold for neural networks.
- Speculative implicit-regularization explanation not validated.
- Only tested on vision models with continuous inputs; not applicable directly to tokenized language models.
Experimental setup
Datasets: ImageNet CIFAR-10 CIFAR-100
Benchmarks: CIFAR-10 adversarial detection benchmark CIFAR-100 adversarial detection benchmark
Metrics: accuracy AUC
Baselines: LID Mahalanobis InputMFS LayerMFS LayerPFS
Builds on: linear mode connectivity feature visualization by optimization Adam percolation theory
Closest papers by meaning
- Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness (ICLR 2020 · similarity 0.89)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.89)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.88)
- Revisiting Mode Connectivity in Neural Networks with Bezier Surface (ICLR 2025 · similarity 0.88)
- On Linear Mode Connectivity of Mixture-of-Experts Architectures (NeurIPS 2025 · similarity 0.85)
- Optimizer-Induced Mode Connectivity: From AdamW to Muon (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.