Research map / Optimization / Distributed and compressed
On Biased Compression for Distributed Learning
Aleksandr Beznosikov, Samuel Horváth, Peter Richtárik, Mher Safaryan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add error feedback to Top-5 compressed SGD (ablation) | Top-5 SGD · VGG19 CIFAR10, 4 nodes | training/test loss and accuracy | Improved |
| use Top-5 (biased) instead of Rand-5 (unbiased) | compressed SGD · VGG19 CIFAR10 | training/test performance | Improved |
| add natural dithering after Top-k | Top-k compression · VGG11 CIFAR10 | communication vs convergence | Improved |
| use natural compression instead of no compression | ALBERT-large · GLUE MRPC | score: 82.5 -> 87.5 | Improved |
| use Power compression instead of no compression | ALBERT-large · GLUE RTE | score: 66.3 -> 46.3 | Hurt |
| use Power compression instead of no compression | ALBERT-large · communication round time | avg time per round: 9.62 -> 1.04 | Improved |
| use natural compression instead of no compression | ALBERT-large · communication round time | avg time per round: 9.62 -> 4.05 | Improved |
| use Top-5 instead of Rand-5 sparsification | Gaussian N(0;1) vectors · information savings d=100 | expected savings: 5 -> 27.14 | Improved |
Closest papers by meaning
- Beyond Communication Overhead: A Multilevel Monte Carlo Approach for Mitigating Compression Bias in Distributed Learning (ICML 2025 · similarity 0.93)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.92)
- Approximate Gradient Coding for Distributed Learning with Heterogeneous Stragglers (NeurIPS 2025 · similarity 0.91)
- Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits (NeurIPS 2026 · similarity 0.91)
- Tight analyses of first-order methods with error feedback (NeurIPS 2025 · similarity 0.89)
- Don't Compress Gradients in Random Reshuffling: Compress Gradient Differences (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
