Research map / Optimization / Implicit bias and generalization
Identifying Equivalent Training Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses Koopman mode decomposition eigenvalues and Wasserstein distance to test topological conjugacy of DNN weight training trajectories, validating on OMD vs OGD and applying to FCN width, CNN early training, and grokking Transformers.
Problem
Existing methods cannot reliably determine whether training dynamics of different DNNs (different architectures, widths, hyperparameters, or initializations) are equivalent, since loss and raw weight trajectories are insufficient and permutation symmetry complicates comparison.
Approach
Treat topological conjugacy as equivalence of Koopman eigenvalues computed from weight trajectories (via time-delay DMD-RRR), compare spectra with Wasserstein distance, and assess significance with a randomized eigenvalue shuffle control; Koopman spectra are invariant to permutation of hidden units.
Key results
- Koopman eigenvalue comparison recovers the known OMD/OGD conjugacy (25% of shuffles not significantly distinct) while OMD/BM and OGD/BM show 0% shuffles exceeding the true distance.
- Narrow (h=5) and wide (h=40) FCNs have non-overlapping Koopman spectra (non-conjugate), while h=10 aligns with h=40.
- LeNet and ResNet-20 early training (first 4 windows of 100 iterations) have statistically significantly different Koopman eigenvalues, indicating non-conjugate early dynamics despite a similar transition timeline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compare Koopman spectra of OMD and OGD | Online mirror descent and online gradient descent · Optimization of the tan objective | Koopman spectral similarity: 25% | No clear effect |
| Compare Koopman spectra with the bisection method | OMD and OGD · Optimization of the tan objective | Koopman spectral similarity: 0% | Hurt |
| Increase hidden width from h=5 to h=40 | Single-hidden-layer FCN · MNIST | Koopman spectral similarity | Hurt |
| Increase hidden width from h=10 to h=40 | Single-hidden-layer FCN · MNIST | Koopman spectral similarity: 81% | No clear effect |
| Use GeLU instead of ReLU (ablation) | Single-hidden-layer FCN · MNIST, varying hidden widths | Koopman spectral similarity | No clear effect |
| Compare first-epoch training windows across LeNet and ResNet-20 | LeNet and ResNet-20 · MNIST and CIFAR-10, early training | Koopman spectral similarity: 2%, 2%, 0%, and 4% | Hurt |
| Compare dynamics over individual epochs rather than 100-iteration windows (ablation) | LeNet and ResNet-20 · MNIST and CIFAR-10, first 20 epochs | Koopman spectral similarity | No clear effect |
| Constrain weight norm during training | Single-hidden-layer Transformer · Modular addition, first 100 training iterations | Koopman spectral similarity | Hurt |
| Constrain weight norm during training | Single-hidden-layer Transformer · Modular addition, early training stability | Training stability | Helped |
| Train FCNs from different random initializations | Single-hidden-layer FCN with h=40 · MNIST | Koopman spectral similarity | Mixed |
Surprising findings
- Transformers with constrained weight norm (no grokking) and unconstrained ones (grokking) have non-conjugate early training dynamics even though their early training losses are similar.
Limitations
- Koopman eigenvalues are approximations from finite data and depend on hyperparameter choices (time-delays, observables, number of modes).
- Magnitude of Wasserstein distance is not clearly linked to degree of dynamical divergence; significance relies on a heuristic shuffle control.
- Theory of conjugacy applies to stable fixed points and discrete spectra; SGD stochasticity makes it an average-dynamics approximation.
Experimental setup
Datasets: MNIST CIFAR-10 modular arithmetic
Benchmarks: MNIST CIFAR-10 modular arithmetic (grokking)
Metrics: Wasserstein distance between Koopman eigenvalues percent of shuffles with distance >= true distance training and test loss test accuracy
Baselines: loss-based comparison randomized shuffle control Frankle et al. (2020) early training timeline
Builds on: Koopman operator theory dynamic mode decomposition DMD-RRR online mirror descent / online gradient descent conjugacy (Ghai et al. 2022) grokking (Power et al.)
Closest papers by meaning
- The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions (ICML 2025 · similarity 0.87)
- Neural Conjugate Flows: A Physics-Informed Architecture with Flow Structure (AAAI 2025 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.87)
- Beyond DSA: Conjugacy-based Comparison of Dynamical Systems (NeurIPS 2026 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.