Research map / Optimization / Optimizers
On the distance between two neural networks and the stability of learning
Jeremy Bernstein, Arash Vahdat, Yisong Yue, Ming-Yu Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Fromage instead of SGD | Fromage · CIFAR-10 train loss | loss: (1.5±0.2)×10^-4 -> (2.5±0.5)×10^-5 | Improved |
| use Fromage instead of Adam | Fromage · CIFAR-10 train loss | loss: (6±3)×10^-5 -> (2.5±0.5)×10^-5 | Improved |
| use Fromage instead of Adam | Fromage · GAN CIFAR-10 | FID: 23.7 -> 16 | Improved |
| use Fromage instead of Adam | Fromage · ImageNet train loss | loss: 2.02 -> 2.001 | No clear effect |
| use Fromage instead of Adam | Fromage · Transformer Wikitext-2 train | perplexity: 36.8 -> 66.1 | Hurt |
| use Fromage instead of SGD | Fromage · Transformer Wikitext-2 train | perplexity: 150.0 -> 66.1 | Improved |
| use Fromage instead of SGD | Fromage · GAN CIFAR-10 | FID: 34 -> 16 | Improved |
| use Fromage instead of off-the-shelf optimisers on very deep MLPs without skip connections | Fromage · MNIST MLP depth over 25 | training accuracy | Improved |
| remove 1/sqrt(1+eta^2) prefactor (LARS-like update) (ablation) | Fromage · spectrally normalised cGAN | numerical stability | Hurt |
Closest papers by meaning
- Loss Landscape Characterization of Neural Networks without Over-Parametrization (NeurIPS 2024 · similarity 0.84)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.84)
- Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.83)
- How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning (ICLR 2025 · similarity 0.83)
- Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss (ICML 2024 · similarity 0.83)
- Function-space Parameterization of Neural Networks for Sequential Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
