Research map / Optimization / Implicit bias and generalization
Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessian
Jack Parker-Holder, Luke Metz, Cinjon Resnick, Hengyuan Hu, Adam Lerer, Alistair Letcher, Alexander Peysakhovich, Aldo Pacchiano, Jakob Foerster
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hessian-eigenvector ridge exploration instead of ERM | Colored MNIST · Colored MNIST test | accuracy: 17.8 \pm 1.33 -> 58.4 \pm 2.41 | Improved |
| Use Hessian-eigenvector ridge exploration instead of ERM | Colored MNIST · Colored MNIST train | accuracy: 87.4 \pm 1.70 -> 65.5 \pm 1.68 | Hurt |
| Use Ridge Rider rather than IRM | Colored MNIST · Colored MNIST | accuracy: 69.7 \pm .710 vs 65.5 \pm 1.68; 65.7 \pm 1.42 vs 58.4 \pm 2.41 | Mixed |
| Update the eigenvector to follow the changing ridge (ablation) | Approximate Ridge Rider · MNIST | test accuracy: approximately 98% vs. beyond 90% | Improved |
| Follow random directions, even when forced to be ascent directions (ablation) | MNIST · MNIST | accuracy: does not exceed 30% | Hurt |
| Follow Hessian eigenvectors rather than random directions | Ridge Rider · Binary-tree RL | percentage of positive solutions found: over 90% vs. at most 50% | Improved |
| Keep the eigenvector fixed rather than updating it along the ridge (ablation) | Ridge Rider · Binary-tree RL, depth 12 | percentage of solutions found | Hurt |
| Use a fixed step size while continuing to update the ridge (ablation) | Ridge Rider · Binary-tree RL, depth 12 | percentage of solutions found | Hurt |
| Use a fixed step size and fixed ridge (ablation) | Ridge Rider · Binary-tree RL, depth 12 | percentage of solutions found | Hurt |
| Use random unit vectors instead of eigenvectors (ablation) | Ridge Rider · Binary-tree RL | percentage of positive solutions found | Hurt |
| Use fixed eigenvectors rather than recomputing them along the path (ablation) | Ridge Rider · Binary-tree RL, depth 12 | percentage of solutions found | No clear effect |
Closest papers by meaning
- An Alternative View: When Does SGD Escape Local Minima? (ICML 2018 · similarity 0.82)
- Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias (NeurIPS 2021 · similarity 0.82)
- SGD Can Converge to Local Maxima (ICLR 2022 · similarity 0.81)
- Learning Curves of Stochastic Gradient Descent in Kernel Regression (ICML 2025 · similarity 0.79)
- A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Bias (NeurIPS 2025 · similarity 0.79)
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (ICLR 2024 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
