Research map / Optimization / Distributed and compressed
Sparsified SGD with Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add error-compensation memory that accumulates compressed-away gradient information | Mem-SGD · strongly convex, smooth finite-sum optimization | convergence rate | Helped |
| use top-k instead of random-k sparsification | Mem-SGD · epsilon and RCV1, sequential | convergence | Helped |
| use top-k instead of random-k sparsification | Parallel-Mem-SGD · multicore shared-memory experiment | scaling | No clear effect |
| reduce the learning-rate delay a to 1 instead of order d/k (ablation) | Mem-SGD · epsilon and RCV1 | convergence | Hurt |
| use top-1 sparsification rather than QSGD quantization | Mem-SGD · epsilon and RCV1 | communicated bits: two orders of magnitude fewer bits | Helped |
| use top-1 sparsification instead of QSGD with 8- or 4-bit quantization | Mem-SGD · epsilon and RCV1 | convergence | No clear effect |
| apply gradient sparsification with top-1 | Mem-SGD · epsilon | communication: 10^3 | Helped |
| use top-10 sparsification | Mem-SGD · RCV1 | communication: approximately an order of magnitude | Helped |
| use multiple cores with sparse updates and independent memories | Parallel-Mem-SGD · multicore shared-memory experiment | speedup | Helped |
| use sparse updates in place of dense lock-free asynchronous updates | Parallel-Mem-SGD · RCV1 multicore shared-memory experiment | scaling | Helped |
Closest papers by meaning
- The Convergence of Sparsified Gradient Methods (NeurIPS 2018 · similarity 0.90)
- Rethinking gradient sparsification as total error minimization (NeurIPS 2021 · similarity 0.86)
- Gradient Sparsification for Communication-Efficient Distributed Optimization (NeurIPS 2018 · similarity 0.86)
- A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication (NeurIPS 2018 · similarity 0.85)
- Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques (NeurIPS 2022 · similarity 0.84)
- MAST: model-agnostic sparsified training (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.