Research map / Optimization / Distributed and compressed
EControl: Fast Distributed Optimization with Compression and Error Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EControl, an error-compensation method for distributed training with contractive compressors that adds a control parameter eta on the feedback signal and a gradient-estimator update, proving convergence in strongly quasi-convex, quasi-convex and nonconvex settings without heterogeneity bounds.
Problem
Error-compensated distributed SGD with contractive compressors is unstable or diverges under data heterogeneity; existing analyses need bounded gradients/dissimilarity, large batches, or unbiased compressors, and lack fast rates in convex regimes.
Approach
Build the error term and the gradient estimator from a single compressed message, and add a control parameter eta that scales the error feedback so compensation is regulated, giving convergence under arbitrary heterogeneity with linear speedup in n.
Key results
- EControl converges on synthetic least squares where Compressed-SGD does not, and its performance is unaffected by heterogeneity parameter zeta while EC degrades as zeta grows.
- On CIFAR-10 with ResNet18 and VGG13, EControl reaches a smaller training loss than EF21 and EF21-SGDM, with test accuracy slightly above EF21 and considerably above EF21-SGDM.
- Theory gives nonconvex complexity LF0 sigma^2/(n eps^2) + LF0 sigma/(delta^2 eps^1.5) + L~F0/(delta eps) with linear speedup in n and no heterogeneity bound.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use error compensation instead of compressed gradients without error compensation | Compressed-SGD · Synthetic least squares | convergence | Helped |
| Use EControl instead of classic EC | Distributed optimization with Top-K compression · Synthetic least squares | error versus communicated bits | Helped |
| Increase the number of clients | EControl · Synthetic least squares | converged error: roughly divided by half | Helped |
| Increase data heterogeneity | EControl · Synthetic least squares | error versus communicated bits | No clear effect |
| Increase data heterogeneity | EC · Synthetic least squares | converged error | Hurt |
| Use EControl instead of SGD with batch size n | Distributed optimization with Top-K compression · Synthetic least squares | communicated bits: roughly by a factor of d/K=10 | Helped |
| Use EControl instead of EF21-SGDM or EF21 | Distributed logistic regression · MNIST logistic regression | training loss | Helped |
| Use EControl instead of EF21-SGDM or EF21 | Distributed logistic regression · MNIST logistic regression | test accuracy: within 1 percent | No clear effect |
| Use EControl instead of EF21 or EF21-SGDM | ResNet18 training · CIFAR-10 ResNet18 | training loss | Helped |
| Use EControl instead of EF21 or EF21-SGDM | ResNet18 training · CIFAR-10 ResNet18 | test accuracy | No clear effect |
| Use EControl instead of EF21 or EF21-SGDM | VGG13 training · CIFAR-10 VGG13 | training loss | Helped |
| Use EControl instead of EF21 and EF21-SGDM | VGG13 training · CIFAR-10 VGG13 | test accuracy | Helped |
Surprising findings
- The EC variant with eta=1 is unstable and sensitive to initialization, motivating the eta control parameter.
Limitations
- Theory requires small stepsizes scaled by delta/L~ and constants such as eta = delta/400 that are conservative.
- Experiments on deep models use a single unspecified setup with stepsizes selected on the test set for CIFAR-10.
Experimental setup
Datasets: MNIST CIFAR-10 synthetic least squares
Benchmarks: MNIST CIFAR-10 synthetic least squares
Metrics: training loss test accuracy suboptimality vs number of bits
Baselines: Compressed-SGD EC EF21 EF21-SGDM mini-batch SGD Choco-SGD
Builds on: error feedback (EC) EF21 DIANA Choco-SGD EF21-SGDM
Closest papers by meaning
- Error Compensated Distributed SGD Can Be Accelerated (NeurIPS 2021 · similarity 0.91)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.89)
- Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits (NeurIPS 2026 · similarity 0.88)
- Tight analyses of first-order methods with error feedback (NeurIPS 2025 · similarity 0.87)
- Distributed Online Convex Optimization with Compressed Communication: Optimal Regret and Applications (NeurIPS 2026 · similarity 0.87)
- Hybrid Decentralized Optimization: Leveraging Both First- and Zeroth-Order Optimizers for Faster Convergence (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.