Research map / Optimization / Distributed and compressed
LASER: Linear Compression in Wireless Distributed Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LASER, a low-rank (PowerSGD-based) gradient compression scheme with error feedback that transmits rank-r factors over noisy wireless multiple-access channels with optimized power allocation, with convergence guarantees close to noiseless SGD.
Problem
Distributed data-parallel SGD over noisy wireless uplink channels suffers from communication bottlenecks; existing compression schemes assume noiseless links or fail on practical deep models.
Approach
Exploit the approximate low-rank structure of gradient matrices: compress via PowerSGD, then transmit the left and right factors over the superposition channel with optimized power split between them, so the server receives a noisy low-rank gradient with variance reduced by roughly the smaller dimension versus sending uncompressed gradients.
Key results
- On WikiText-103 GPT-2, LASER reaches perplexity 76 vs 212 for Z-SGD at low power (P=10K), a 64% improvement; at moderate power 35 vs 71 (50%).
- LASER requires 16x less power than Z-SGD to reach target accuracies up to 91% on CIFAR-10.
- LASER sends 3 MB per iteration vs 496 MB for Z-SGD on WikiText-103 (165x reduction).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use LASER instead of Z-SGD | GPT-2 · WikiText-103, P=10 K | perplexity: 212 -> 76; 64% improvement | Helped |
| Use LASER instead of Z-SGD | GPT-2 · WikiText-103, P=160 K | perplexity: 71 -> 35; 50% | Helped |
| Use LASER instead of Z-SGD | GPT-2 · WikiText-103, very high power | perplexity | No clear effect |
| Use LASER instead of Z-SGD | ResNet18 · CIFAR-10, target accuracy through 91% | power required: 16× less power | Helped |
| Use LASER instead of Z-SGD | GPT-2 · WikiText-103 | data sent per iteration: 496 MB -> 3 MB; 165× | Helped |
| Use LASER instead of Random-K | 1-layer NN · MNIST, P=0.1 | test accuracy: 86.1% -> 84.3% | Hurt |
| Use LASER instead of the other algorithms | 1-layer NN · MNIST, P=1 and P=10 | test accuracy: 89.9% at P=1; 92.3% at P=10 | Helped |
| Use A-DSGD instead of simpler compression schemes | 1-layer NN · MNIST, all tested power settings | test accuracy | Hurt |
| Use a constant power policy instead of time-varying policies (ablation) | LASER · CIFAR-10 | accuracy | Helped |
| Vary the power policy over time instead of using constant power (ablation) | LASER · CIFAR-10 | accuracy | No clear effect |
| Set baseline compression factor to 0.2 (ablation) | compression algorithms · MNIST, CIFAR-10, CIFAR-100, and WikiText-103 | performance: 0.2 | Helped |
Surprising findings
- Constant power policy performs best for both LASER and Z-SGD compared to time-varying power schedules.
- Random-K slightly beats LASER in the very noisy MNIST regime (P=0.1).
Limitations
- Theory assumes a single client (k=1) and extends to multiple clients by heuristic scaling; uplink-only noise modeled, with downlink noise and fast fading left to future work.
- Channel is simulated with additive Gaussian noise; fast fading without CSI not covered.
- A-DSGD does not scale beyond MNIST; some baselines omitted for scale or convergence reasons.
Experimental setup
Datasets: WikiText-103 CIFAR-10 CIFAR-100 MNIST
Benchmarks: WikiText-103 CIFAR-10 CIFAR-100 MNIST
Metrics: perplexity top-1 accuracy power required to reach target communication cost (MB per iteration)
Baselines: Z-SGD Signum Random-K Sketching A-DSGD Noiseless SGD
Builds on: PowerSGD error feedback SGD A-DSGD
Closest papers by meaning
- PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization (NeurIPS 2019 · similarity 0.91)
- Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits (NeurIPS 2026 · similarity 0.87)
- Communication-efficient Distributed Learning for Large Batch Optimization (ICML 2022 · similarity 0.87)
- Towards Faster Decentralized Stochastic Optimization with Communication Compression (ICLR 2025 · similarity 0.87)
- Accelerated Methods with Compressed Communications for Distributed Optimization Problems Under Data Similarity (AAAI 2025 · similarity 0.87)
- Error Compensated Distributed SGD Can Be Accelerated (NeurIPS 2021 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.