Research map / Architectures and efficiency / RNNs and normalization
Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies factors behind DNN valley asymmetry and finds that sign consistency between the 1D perturbation noise and the converged weights determines asymmetry, with theory from ReLU and softmax and applications to model soups and a sign-regularized federated learning method (FedSign).
Problem
The asymmetry of loss valleys around converged DNN solutions has been observed but not systematically studied; it is unclear what causes it and how it matters for generalization and model fusion.
Approach
Valley symmetry along 1D interpolation is governed by the degree of sign consistency between the noise direction and the converged parameters; positive-direction interpolation along sign-consistent noise stays flat while the negative direction sharpens. This is explained via ReLU activation persistence and softmax Hessian trace, and used to explain model soups and to regularize parameter signs in
Key results
- FedSign reaches 82.59% on CIFAR-10 with Dir alpha=10 versus 81.53% for FedAvg, and 74.59% vs 72.19% on CINIC-10 with alpha=1.0.
- Interpolation accuracy of models fine-tuned from a pre-trained ResNet18 on Flowers exceeds individual models, with sign consistency ratio above 0.95, while randomly initialized VGG16BN on CIFAR10 shows a significant interpolation barrier at epoch 50.
- Traditional BN initialization (all ones) yields all-positive converged BN weights and asymmetric valleys, whereas Gaussian BN initialization gives nearly symmetric valleys.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CIFAR-10, Dir. α=10.0 | accuracy: 82.44 -> 82.59 | Helped |
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CIFAR-10, Dir. α=1.0 | accuracy: 80.54 ± 0.11 -> 80.76 ± 0.14 | Helped |
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CIFAR-10, Dir. α=0.5 | accuracy: 78.12 -> 78.41 | Helped |
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CINIC-10, Dir. α=10.0 | accuracy: 76.84 -> 77.05 | Helped |
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CINIC-10, Dir. α=1.0 | accuracy: 73.46 -> 74.59 | Helped |
| regularize local parameter sign changes with FedSign instead of using the strongest listed baseline | federated learning · CINIC-10, Dir. α=0.5 | accuracy: 70.14 -> 70.63 | Helped |
| replace noise signs with the converged model's parameter signs (ablation) | VGG16 with BN on CIFAR-10 · CIFAR-10 | valley asymmetry | Helped |
| increase the fraction of Gaussian-noise elements whose signs match the converged parameters (ablation) | VGG16 with BN and other tested networks · CIFAR-10 and other tested datasets | positive-versus-negative interpolation test error | Mixed |
| use initialization, converged parameters, parameter signs, or centered parameter signs as the noise direction (ablation) | VGG16 with BN on CIFAR-10 · CIFAR-10 | valley asymmetry | Mixed |
| initialize BN weights from G(0, 0.1) rather than all ones or U(0, 1) (ablation) | VGG16 with BN on CIFAR-10 · CIFAR-10 | valley asymmetry | Hurt |
| vary learning rate, batch size, or weight decay (ablation) | VGG16 with BN on CIFAR-10 · CIFAR-10 | valley symmetry | No clear effect |
| use larger learning rate, smaller batch size, or larger weight decay (ablation) | VGG16 with BN and ResNet20 · CIFAR-10 | test error | Helped |
Surprising findings
- Changing the sign of common noise (e.g., Gaussian) to match the converged weights makes valleys completely asymmetric, while the same noise without sign change is symmetric.
- VGG11 without BN shows no asymmetry even with the initialization as noise, indicating BN initialization matters for asymmetry.
Limitations
- No formal proofs of the conditions under which asymmetric valleys arise.
- Only image classification tasks are studied.
- Theoretical analysis relies on simplifying assumptions (e.g., w = a*h + Gaussian noise).
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN CINIC10 Flowers Food101 ImageNet sklearn.digits
Benchmarks: CIFAR10 CIFAR100 SVHN CINIC10 Food101 Flowers ImageNet
Metrics: test error cross-entropy loss test accuracy
Baselines: FedAvg FedProx MOON FedDyn FedPAN
Builds on: filter normalization adaptive sharpness asymmetric valley model soups FedAvg
Closest papers by meaning
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima (NeurIPS 2019 · similarity 0.90)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.85)
- Symmetry Induces Structure and Constraint of Learning (ICML 2024 · similarity 0.85)
- Generalized Linear Mode Connectivity for Transformers (NeurIPS 2025 · similarity 0.85)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.85)
- The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.