Research map / Optimization / Distributed and compressed
Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EF-Landing, a retraction-free, communication-compressed distributed stochastic optimization method on the Stiefel manifold that compresses Euclidean gradients with momentum error feedback, with linear-speedup convergence analysis and a block-wise extension.
Problem
Distributed stochastic optimization on the Stiefel manifold (orthonormal-column constraints) where communication of full gradient matrices is costly and retractions are expensive.
Approach
Compress the Euclidean gradient (not the Landing descent direction) so that the orthogonality between the Riemannian gradient and penalty gradient is preserved, and add momentum-based error feedback to avoid stagnation from vanilla greedy compression; combine with gradient clipping to stay in the safe region.
Key results
- On distributed online PCA (p=1000), EF-Landing with Top-K, Rand-K and QSGD reaches the same training loss as vanilla Landing while significantly reducing communication.
- On CIFAR-10 with ResNet-18 (Top-K/Rand-K retention 0.2), EF-Landing matches vanilla Landing and QR retraction accuracy per iteration with reduced communication.
- Theory gives O(1/sqrt(NK)) stochastic rate with linear speedup, matching uncompressed stochastic Landing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add error feedback to vanilla gradient compression | Landing on the Stiefel manifold · Deterministic single-node Stiefel optimization | convergence to a stationary solution | Helped |
| use vanilla gradient compression without error feedback (ablation) | Landing on the Stiefel manifold · Deterministic single-node Stiefel optimization | convergence to a stationary solution | Hurt |
| add communication compression with error feedback | Landing on the Stiefel manifold · Deterministic Stiefel optimization | convergence rate: \(\mathcal{O}(1/K)\) | No clear effect |
| add communication compression with error feedback | Stochastic Landing · Stochastic Stiefel optimization | convergence rate: \(\mathcal{O}(1/\sqrt{NK})\) | No clear effect |
| increase communication compression aggressiveness (ablation) | EF-Landing · Deterministic Stiefel optimization | convergence speed | Hurt |
| use error feedback and gradient compression | EF-Landing · Distributed online PCA | training loss | No clear effect |
| use error feedback and gradient compression | EF-Landing · ResNet-18 on CIFAR-10 | accuracy | No clear effect |
| use a penalty method instead of EF-Landing (ablation) | Distributed online PCA · Distributed online PCA | training loss | Hurt |
Surprising findings
- Vanilla gradient compression can cause stagnation on the Stiefel manifold even in a deterministic single-node setting, unlike unconstrained optimization.
Limitations
- Requires a global gradient bound L' and gradient clipping to stay in the safe region.
- More aggressive compression slows convergence (theta to 0).
- Experiments limited to online PCA and ResNet-18/VGG16 on small image datasets.
Experimental setup
Datasets: CIFAR-10 MNIST synthetic online PCA data
Benchmarks: CIFAR-10 MNIST synthetic distributed online PCA
Metrics: training loss f(X)-f(X*) constraint violation N(X) test accuracy communication overhead
Baselines: vanilla Landing QR retraction penalty method decentralized training frameworks centralized multi-local-update methods
Builds on: Landing algorithm error feedback momentum error feedback Top-K QSGD
Closest papers by meaning
- Optimization without Retraction on the Random Generalized Stiefel Manifold (ICML 2024 · similarity 0.86)
- Towards Faster Decentralized Stochastic Optimization with Communication Compression (ICLR 2025 · similarity 0.86)
- Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits (NeurIPS 2026 · similarity 0.86)
- Federated Optimization with Doubly Regularized Drift Correction (ICML 2024 · similarity 0.85)
- Safe-EF: Error Feedback for Non-smooth Constrained Optimization (ICML 2025 · similarity 0.85)
- Decentralized Projected Riemannian Stochastic Recursive Momentum Method for Nonconvex Optimization (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.