Research map / Optimization / Distributed and compressed
Error Feedback Fixes SignSGD and other Gradient Compression Schemes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-100, Resnet18, batch size 128 | generalization gap (test accuracy percentage points vs. SGDM): -2.21 -> -0.92 | Helped |
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-100, Resnet18, batch size 32 | generalization gap (test accuracy percentage points vs. SGDM): -3.04 -> -0.79 | Helped |
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-100, Resnet18, batch size 8 | generalization gap (test accuracy percentage points vs. SGDM): -36.35 -> -0.64 | Helped |
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-10, VGG19, batch size 128 | generalization gap (test accuracy percentage points vs. SGDM): -1.31 -> -0.68 | Helped |
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-10, VGG19, batch size 32 | generalization gap (test accuracy percentage points vs. SGDM): -1.49 -> -0.71 | Helped |
| Add error feedback to scaled SignSGD (ablation) | Scaled SignSGD · CIFAR-10, VGG19, batch size 8 | generalization gap (test accuracy percentage points vs. SGDM): -20.22 -> -0.27 | Helped |
| Use SignSGD without error feedback rather than Error-Feedback SignSGD (ablation) | Error-Feedback SignSGD · CIFAR-100 and CIFAR-10, smaller batch sizes | generalization gap (test accuracy percentage points vs. SGDM): -0.64 vs. -36.35 on CIFAR-100; -0.27 vs. -20.22 on CIFAR-10, batch size 8 | Hurt |
| Use sparse-noise SignSGD instead of Error-Feedback SignSGD | Error-Feedback SignSGD · Quadratic toy problem with sparse noise | convergence rate | Mixed |
| Use error feedback with a compressed gradient rather than naive SignSGD | SignSGD · Convex counterexamples and smooth non-convex optimization | convergence | Helped |
Closest papers by meaning
- signSGD: Compressed Optimisation for Non-Convex Problems (ICML 2018 · similarity 0.93)
- Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction (NeurIPS 2024 · similarity 0.90)
- When and Why SignSGD Outperforms SGD: A Theoretical Study Based on $\ell_1$-norm Lower Bounds (NeurIPS 2026 · similarity 0.89)
- Exact risk curves of signSGD in High-Dimensions: quantifying preconditioning and noise-compression effects (ICML 2025 · similarity 0.88)
- z-SignFedAvg: A Unified Stochastic Sign-Based Compression for Federated Learning (AAAI 2024 · similarity 0.87)
- SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.