Research map / Optimization / Implicit bias and generalization
Improving Convergence and Generalization Using Parameter Symmetries
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that symmetry teleportation provably accelerates SGD convergence, relates minimum curvature to generalization, and extends teleportation to standard optimizers and learned meta-optimizers.
Problem
Teleportation via parameter space symmetries accelerates optimization empirically, but its mechanism and convergence guarantees are not well understood, and its potential for improving generalization is unexplored.
Approach
Teleport parameters within loss level sets to points of maximal gradient norm, with convergence guarantees for SGD and a novel curvature measure of minima (via curves generated by Lie group actions) that is linked to generalization and used as a teleportation objective; also learn teleportation jointly with an LSTM meta-optimizer.
Key results
- Pearson correlation with validation loss is strong for sharpness (0.704 MNIST, 0.790 Fashion-MNIST, 0.899 CIFAR-10) while curvature correlations are weakly negative (-0.050, -0.232, -0.167).
- Teleportation improves convergence rate per epoch for AdaGrad, SGD with momentum, RMSProp, and Adam on MNIST, with teleport runtime smaller than one epoch of training.
- Teleporting to higher-curvature minima lowers validation loss on CIFAR-10, while teleporting to lower-curvature minima has the opposite effect; sharpness-based teleportation only slightly helps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Teleport to points with larger curvature | SGD-trained MLP · CIFAR-10 | validation loss | Helped |
| Teleport to points with smaller curvature | SGD-trained MLP · CIFAR-10 | validation loss | Hurt |
| Teleport to flatter points by decreasing sharpness | SGD-trained MLP · CIFAR-10 | validation loss | Helped |
| Teleport to sharper points by increasing sharpness | SGD-trained MLP · CIFAR-10 | validation loss | No clear effect |
| Add teleportation before optimization | AdaGrad, SGD with momentum, RMSProp, and Adam · MNIST | convergence rate | Helped |
| Learn both the local update and teleportation | LSTM meta-optimizer · Two-layer LeakyReLU network optimization | convergence rate | Helped |
| Remove teleportation and learn only the local update (ablation) | LSTM(update,tele) meta-optimizer · Two-layer LeakyReLU network optimization | convergence rate | Hurt |
| Remove the local update and learn teleportation and learning rate only (ablation) | LSTM(update,tele) meta-optimizer · Two-layer LeakyReLU network optimization | convergence rate | Hurt |
Surprising findings
- Teleporting to flatter points gives only negligible validation loss improvement, and sharper-point teleportation has no effect.
- Curvature of minima is negatively correlated with validation loss for the studied architecture, even though sharpness is strongly positively correlated.
Limitations
- Theoretical guarantees rely on assumptions such as smoothness and strict convexity for some results; one teleportation optimality needs a specific sufficient condition.
- Curvature-generalization link is empirical and locally observed; theoretical origin unclear.
- Teleportation before every step is computationally expensive, so it is applied only occasionally.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10
Benchmarks: MNIST Fashion-MNIST CIFAR-10
Metrics: test loss training loss Pearson correlation with validation loss convergence rate per epoch
Baselines: SGD AdaGrad SGD with momentum RMSProp Adam LSTM(update) LSTM(lr,tele) GD
Builds on: symmetry teleportation learning to learn by gradient descent by gradient descent
Closest papers by meaning
- Continual Optimization with Symmetry Teleportation for Multi-Task Learning (NeurIPS 2025 · similarity 0.88)
- Global curvature for second-order optimization of neural networks (ICML 2025 · similarity 0.86)
- A geometric framework for momentum-based optimizers for low-rank training (NeurIPS 2025 · similarity 0.86)
- Scalable Decentralized Learning with Teleportation (ICLR 2025 · similarity 0.86)
- GradMetaNet: An Equivariant Architecture for Learning on Gradients (NeurIPS 2025 · similarity 0.85)
- Understanding Mode Connectivity via Parameter Space Symmetry (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.