Research map / Optimization / Distributed and compressed
Det-CGD: Compressed Gradient Descent with Matrix Stepsizes for Non-Convex Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Det-CGD, compressed/sketched gradient descent with a fixed matrix stepsize for matrix-smooth non-convex objectives, with single-node and distributed analyses and layer-wise Bernoulli sketches giving "free" compression.
Problem
Non-convex smooth optimization with compressed gradients where the smoothness structure (matrix smoothness, especially block-diagonal layer-wise structure of neural networks) is not exploited by scalar stepsizes.
Approach
Replace the scalar stepsize in compressed gradient descent with a fixed positive-definite matrix stepsize D, with two sketch-and-stepsize orderings (D S grad and T D grad), derive convergence under matrix smoothness, optimize D via convex programs, and design layer-wise Bernoulli sketches that compress without losing communication complexity.
Key results
- Matrix-stepsize sketch gradient descent improves on GD and scalar-stepsize DCGD in iteration and communication complexity under matrix smoothness.
- Bernoulli layer-wise sketches achieve communication complexity d·det(L)^(1/d), matching uncompressed matrix-stepsize GD while reducing bits per iteration by q when layer widths are equal.
- Distributed theorem recovers the scalar DCGD result and improves it in general, with a neighborhood term that vanishes as K grows under the stated stepsize conditions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a matrix stepsize without compression instead of scalar-stepsize GD | det-CGD1 · General structure | communication complexity: d \cdot \lambda_{\max}(\mL) -> d \cdot \det(\mL)^{1/d} | Helped |
| Use a matrix stepsize without compression instead of scalar-stepsize GD | det-CGD2 · General structure | communication complexity: d \cdot \lambda_{\max}(\mL) -> d \cdot \det(\mL)^{1/d} | Helped |
| Use Bernoulli-q layer sampling instead of uniform rand-k sampling | det-CGD2 · Layer-wise structure | communication complexity: d | Helped |
| Use Bernoulli-q layer sampling while reducing bits sent per iteration | det-CGD2 · Layer-wise structure | communication complexity: d\cdot\det(\mL)^{1/d} | No clear effect |
| Use rand-1 with diagonal inverse-smoothness stepsize instead of inverse-smoothness stepsize | det-CGD1 · General structure | communication complexity: row 5 -> row 7 | Helped |
| Use rand-1 with inverse-square-root stepsize instead of diagonal inverse-smoothness stepsize | det-CGD1 · General structure | communication complexity: 4; 8; 4; \simeq 3.88 | Mixed |
Surprising findings
- Compression can be free: Bernoulli layer sketches match the communication complexity of uncompressed matrix-stepsize GD.
- Rows 5 and 6 of the complexity table are not uniformly comparable, and the normalized-determinant metric is needed to compare stepsizes.
Limitations
- Gradient-norm guarantee is in a D-weighted norm, which does not imply Euclidean stationarity.
- Matrix stepsize optimization is a convex program that is not directly DCP-compliant and needs extra reformulation for CVXPY.
- Experiments are simple and the analysis is limited to fixed matrix stepsizes and specific sketches.
Experimental setup
Metrics: communication complexity iteration complexity gradient norm
Baselines: GD scalar-stepsize DCGD Rand-k compression Newton Star
Builds on: DCGD gradient descent Newton Star
Closest papers by meaning
- Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization (ICML 2020 · similarity 0.87)
- Block Acceleration Without Momentum: On Optimal Stepsizes of Block Gradient Descent for Least-Squares (ICML 2024 · similarity 0.87)
- Gradient Compressed Sensing: A Query-Efficient Gradient Estimator for High-Dimensional Zeroth-Order Optimization (ICML 2024 · similarity 0.86)
- Provable Acceleration of Nesterov's Accelerated Gradient for Asymmetric Matrix Factorization and Linear Neural Networks (NeurIPS 2024 · similarity 0.86)
- Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits (NeurIPS 2026 · similarity 0.86)
- Adaptive Proximal Gradient Method for Convex Optimization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.