Research map / Optimization / Nonconvex and smooth optimization
Learning-Rate-Free Stochastic Optimization over Riemannian Manifolds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes learning-rate-free Riemannian stochastic optimizers (RDoG, NRDoG, RDoWG, T-RDoWG) that replace the step size with distance-over-gradients estimates, with high-probability convergence guarantees and experiments on sphere, Grassmann PCA, and Poincaré embeddings.
Problem
Riemannian stochastic gradient methods require careful learning-rate tuning, and no learning-rate-free methods existed for geodesically convex optimization on Riemannian manifolds.
Approach
Extends Euclidean Distance over Gradients (DoG) and Distance over Weighted Gradients (DoWG) to Riemannian manifolds by estimating the unknown distance to the optimum with the maximum distance from the initial point, using a trigonometric distance bound with a curvature function to set step sizes automatically.
Key results
- On Rayleigh quotient maximization, RDoG, RDoWG and NRDoG remain robust across initial distance estimates from 1e-8 to 1, while RSGD and RAdam depend on the learning rate.
- On Grassmann PCA, RDoG, RDoWG and NRDoG are insensitive to the initial distance estimate, while RSGD and RAdam need a narrow range of tuned learning rates.
- On Poincaré embedding of the WordNet mammal subtree, RDoG and RDoWG show competitive mAP without a burn-in heuristic, whereas RSGD and RAdam degrade without burn-in.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RDoG, RDoWG, or NRDoG instead of learning-rate-tuned RSGD or RADAM | Riemannian optimizers · Rayleigh quotient maximization on the sphere | geodesic distance to numerical solution | Helped |
| Use RDoG instead of learning-rate-tuned RSGD | Riemannian stochastic optimization · Rayleigh quotient maximization on the sphere | regret | Helped |
| Use RDoG, RDoWG, or NRDoG instead of learning-rate-tuned RSGD or RADAM | Riemannian optimizers · PCA on the Grassmann manifold (Wine, Waveform-5000, Tiny ImageNet) | geodesic distance to numerical solution | Helped |
| Omit the burn-in heuristic (ablation) | RSGD and RADAM · WordNet mammals' subtree embedding in the Poincaré ball | mean average precision | Hurt |
| Use RDoG or RDoWG instead of learning-rate-dependent optimizers | Riemannian optimizers · WordNet mammals' subtree embedding in the Poincaré ball | mean average precision | No clear effect |
Limitations
- Theory assumes bounded iterates for the main RDoG and RDoWG guarantees; stability requires the tamed variant T-RDoWG with polylogarithmic smaller steps.
- Requires computable exponential map and geodesic distance; retractions and manifolds like Stiefel are not handled.
- Experiments are limited to small-scale problems on a laptop.
Experimental setup
Datasets: Wine Waveform-5000 Tiny ImageNet WordNet mammal subtree
Benchmarks: Rayleigh quotient on sphere (d=1000) PCA on Grassmann manifold (Wine, Waveform-5000, Tiny ImageNet) Poincaré embedding of WordNet mammals
Metrics: geodesic distance to numerical solution regret mean average precision
Baselines: RSGD RAdam RDoG RDoWG NRDoG
Builds on: Distance over Gradients (DoG) Distance over Weighted Gradients (DoWG) Riemannian stochastic gradient descent RAdam
Closest papers by meaning
- Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method (ICML 2025 · similarity 0.90)
- Finite-Time Analysis of Stochastic Nonconvex Nonsmooth Optimization on the Riemannian Manifolds (NeurIPS 2025 · similarity 0.89)
- Riemannian coordinate descent algorithms on matrix manifolds (ICML 2024 · similarity 0.89)
- A Framework for Bilevel Optimization on Riemannian Manifolds (NeurIPS 2024 · similarity 0.88)
- Implicit Riemannian Optimism with Applications to Min-Max Problems (ICML 2025 · similarity 0.88)
- SERENA: A Unified Stochastic Recursive Variance Reduced Gradient Framework for Riemannian Non-Convex Optimization (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.