Research map / Optimization / Optimizers
Better Training using Weight-Constrained Stochastic Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add circle constraints using c-CoLud | Transformer · Penn Treebank | validation loss: 4.87 -> 4.81 | Helped |
| Add circle constraints using c-CoLud | Transformer · Wikitext-2 | validation loss: 5.13 -> 5.09 | Helped |
| Add magnitude constraints and stochastic noise | SGD training of a single-hidden-layer perceptron · 4-turn spiral | test accuracy: 81% -> 91.7% | Helped |
| Add magnitude constraints and stochastic noise | SGD training of a single-hidden-layer perceptron · 4-turn spiral | mean curvature: 519 -> 6.08 | Helped |
| Add magnitude constraints | SGD training of a single-hidden-layer perceptron · 4-turn spiral | mean curvature: 519 -> 9.38 | Helped |
| Use weight decay instead of magnitude constraints (ablation) | SGD training of a single-hidden-layer perceptron · 4-turn spiral | test accuracy | No clear effect |
| Replace standard initialization with orthogonal initialization (ablation) | SGD-trained MLP · 4-turn spiral | test accuracy | Mixed |
| Replace hard orthogonality constraints with orthogonal regularization (ablation) | MLP · 4-turn spiral | test accuracy | Hurt |
| Add orthogonality constraints with o-CoLod | MLP · 4-turn spiral | test accuracy | Helped |
| Add a small temperature perturbation (ablation) | o-CoLod · 4-turn spiral | test accuracy | Helped |
| Add orthogonality constraints with o-CoLud and omit weight decay | ResNet-34 · CIFAR-10 | test accuracy | Helped |
| Add circle constraints with c-CoLud | 1000-node single-hidden-layer perceptron · Fashion-MNIST | test accuracy | Helped |
Closest papers by meaning
- Langevin Dynamics with Continuous Tempering for Training Deep Neural Networks (NeurIPS 2017 · similarity 0.89)
- Explicitly Imposing Constraints in Deep Networks via Conditional Gradients Gives Improved Generalization and Faster Convergence (AAAI 2019 · similarity 0.88)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.88)
- Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks (AAAI 2016 · similarity 0.88)
- Improving Deep Learning Optimization through Constrained Parameter Regularization (NeurIPS 2024 · similarity 0.88)
- No Wrong Turns: The Simple Geometry Of Neural Networks Optimization Paths (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.