Research map / Architectures and efficiency / RNNs and normalization
Enhancing Neural Training via a Correlated Dynamics Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Correlation Mode Decomposition (CMD), which clusters correlated neural-network weight trajectories into a few modes with reference trajectories and affine coefficients, enabling online dynamics modeling, embedded training, and communication-reduced federated learning.
Problem
Modeling neural network training dynamics compactly and efficiently, since full weight trajectories are costly to store and analyze, and existing low-dimensional or DMD-based methods do not capture the complex dynamics well.
Approach
Parameter trajectories during training are highly correlated over time; clustering them by absolute correlation into modes, each represented by one reference trajectory with per-weight affine coefficients, models dynamics with few modes. An iterative (online) variant updates the affine coefficients each epoch without storing trajectories, and gradually embeds stable weights into the model.
Key results
- Online CMD reaches 92.43% ± 0.34 test accuracy on ResNet18/CIFAR10 vs 90.91% ± 0.52 for baseline SGD and 91.03% ± 0.38 for P-BFGS (long warm-up).
- Online CMD outperforms P-BFGS on all five CIFAR10 CNN architectures and ViT-b-16 (97.99% vs 95.05% for equal warm-up).
- Gradual CMD embedding yields roughly 50% reduction in total trained parameters with improved accuracy in the reported setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace baseline SGD with Online CMD | image classifiers · CIFAR10 test, ResNet18, WideRes, PreRes164, LeNet-5, GoogleNet, and ViT-b-16 | accuracy: 90.91% ± 0.52 -> 92.43% ± 0.34; 92.70% ± 0.48 -> 93.21% ± 0.50; 90.64% ± 0.98 -> 91.34% ± 1.20; 75.36% ± 0.57 -> 76.77% | Helped |
| use Online CMD rather than the strongest listed P-BFGS result | image classifiers · CIFAR10 test, ResNet18, WideRes, PreRes164, LeNet-5, and GoogleNet | accuracy: 89.34% ± 0.54 -> 92.43% ± 0.34; 92.78% ± 0.25 -> 93.21% ± 0.50; 90.91% ± 0.77 -> 91.34% ± 1.20; 76.44% ± 0.37 -> 76.77% | Helped |
| apply Online CMD rather than SWA or baseline SGD smoothing comparison | ResNet18 · CIFAR10 test; SGD, SGD + Mom, SGD + Mom + SCH, and Adam | accuracy: 92.09% ± 0.10 -> 92.26% ± 0.12; 92.28% ± 0.44 -> 92.43% ± 0.34; 91.29% ± 0.23 -> 91.68% ± 0.21; 91.65% ± 0.43 -> 91.70% | Helped |
| replace post-hoc CMD with online CMD (ablation) | ResNet18 CMD at 1, 2, 5, and 10 modes · CIFAR10 test | accuracy: 92.06% ± 0.02 -> 92.06% ± 0.05; 92.01% ± 0.04 -> 92.07% ± 0.07; 91.96% ± 0.05 -> 91.94% ± 0.12; 92.05% ± 0.25 -> 91.98% | No clear effect |
| vary the number of CMD modes (ablation) | ResNet18 · CIFAR10 test | accuracy | No clear effect |
| vary the number of sampled weights K (ablation) | ResNet18 CMD with M=10 · CIFAR10 test | accuracy | No clear effect |
| use different random subsets of sampled weights (ablation) | ResNet18 CMD with M=10 and K=1000 · CIFAR10 test | accuracy | Helped |
| change random weight initialization (ablation) | ResNet18 CMD with M=10 and K=1000 · CIFAR10 test | accuracy | No clear effect |
| train only reference weights after warm-up instead of gradual CMD embedding (ablation) | ResNet18 · CIFAR10 test | accuracy: 91.50% -> 89.49% | Hurt |
| gradually embed stable weights into CMD dynamics (ablation) | ResNet18 · CIFAR10 test | accuracy: 91.24% -> 91.50% | Helped |
| assign CMD-modeled weights periodically, with SGD updates between assignments (ablation) | CMD training · training performance across epochs | accuracy | Hurt |
| use CMD embedding instantaneously rather than gradually (ablation) | CMD training · training stability | stability | Hurt |
Surprising findings
- Test accuracy of CMD models is close to invariant to the number of modes, even though weight reconstruction MSE decreases with more modes.
- Naive assignment of CMD-modeled weights periodically or training only reference weights post warm-up degrades performance back toward SGD or below it.
Limitations
- Post-hoc CMD requires storing the full training trajectory, which is resource-intensive.
- Federated learning evaluation is described as preliminary and without additional compression.
- P-BFGS could not be run with long warm-up or on ViT-b-16 on the authors' hardware, limiting comparisons.
Experimental setup
Datasets: CIFAR10 MNIST PASCAL VOC 2012 CelebA-HQ
Benchmarks: CIFAR10 PASCAL VOC 2012 CelebA-HQ
Metrics: test accuracy pixel accuracy mIoU LPIPS FID communication volume
Baselines: P-BFGS APF A-APF EMA SWA DMD Baseline SGD
Builds on: P-BFGS low-dimensional training subspace dynamic mode decomposition APF adaptive parameter freezing
Closest papers by meaning
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)
- When Graph Neural Networks Meet Dynamic Mode Decomposition (ICLR 2025 · similarity 0.86)
- Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.86)
- Identifying Equivalent Training Dynamics (NeurIPS 2024 · similarity 0.85)
- Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.