Research map / Architectures and efficiency / RNNs and normalization
Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Subspace Networks, a model-parallel activation and activation-gradient compression scheme that constrains projection-matrix row spaces to a shared low-rank subspace, enabling lossless 100x compression for decentralized pipeline training.
Problem
Decentralized training is bottlenecked by communication in model parallelism, where activations and activation gradients must be passed between layers; existing lossy compression methods designed for data-parallel weight gradients accumulate errors and fail to converge in this setting.
Approach
Explicitly constrain transformer projection-matrix rows to a shared low-dimensional subspace S, decompose token and positional embeddings into fixed high-rank and trainable low-rank parts, so residual activations and activation gradients can be compressed losslessly to k of d dimensions; subspace is updated on the Grassmann manifold and AdamW is modified to keep row-wise adaptive scaling so update
Key results
- On 8-layer 2B model, compressed decentralized run at 80Mbps reaches OWT perplexity 46.75 vs 47.22 for centralized 100Gbps, while uncompressed 80Mbps decentralized gets 925.19.
- Compressed decentralized 8B LLaMA over 4 regions matches centralized wall-clock convergence, while uncompressed decentralized was 13x slower.
- At 100x compression, TopK, SVD low-rank, and quantization fail to converge on WikiText, while the proposed method matches the uncompressed baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compress pipeline-parallel activations and activation gradients with Subspace Networks | LLaMA-based 2B model · 80Mbps decentralized training on OpenWebText | perplexity: 925.19 -> 46.75 | Helped |
| Compress pipeline-parallel activations and activation gradients with Subspace Networks | LLaMA-based 2B model · 80Mbps decentralized training on BookCorpus | perplexity: 108.85 -> 17.63 | Helped |
| Compress pipeline-parallel activations and activation gradients with Subspace Networks | LLaMA-based 2B model · 80Mbps decentralized training on WikiText | perplexity: 601.84 -> 23.01 | Helped |
| Compress pipeline-parallel communication with Subspace Networks | LLaMA-based 2B model · 80Mbps decentralized training | tokens per second: 36.12 -> 592.41 | Helped |
| Use Subspace Networks compression instead of uncompressed centralized training | 640M model · C4 validation at 12 billion tokens | perplexity: 12.61 -> 12.53 | No clear effect |
| Use Subspace Networks compression instead of uncompressed centralized training | 640M model · BookCorpus validation at 12 billion tokens | perplexity: 12.79 -> 12.67 | No clear effect |
| Use Top-k, quantization, or low-rank SVD compression at 100x (ablation) | 8-layer model · WikiText | convergence | Hurt |
| Restrict the token embedding table to the shared low-dimensional subspace (ablation) | Subspace Networks · Transformer training | performance | Hurt |
| Allow the shared subspace to drift using Grassmann updates (ablation) | Subspace Networks · Transformer training | convergence | Helped |
| Increase model depth from 8 to 16 layers (ablation) | Subspace Networks · C4 | convergence relative to centralized baseline | No clear effect |
| Assign two layers per GPU rather than one in the 16-layer setup (ablation) | Decentralized Subspace Networks · C4, 16-layer model | performance gap to centralized training | Helped |
| Use Subspace Networks compression for inference | 2B model · 80Mbps inference | speedup: almost a 100x speedup | Helped |
Surprising findings
- The compressed decentralized model at 80Mbps slightly outperforms the centralized 100Gbps model in perplexity at equal wall-clock time.
- Activation compression did not degrade convergence even as depth increased from 8 to 16 layers and to 32 layers.
Limitations
- Lossless claim relies on projection-matrix constraints; the W_p1 projection is still applied iteratively due to nonlinearity, so exactness is not fully maintained across all weights.
- Method benefits depend on slower accelerators; effectiveness is reported to diminish with faster GPUs.
- Evaluation focuses on small models (2B-8B) and perplexity/loss, with no downstream task benchmarks reported.
Experimental setup
Datasets: WikiText BookCorpus OpenWebText C4
Benchmarks: WikiText BookCorpus OpenWebText C4
Metrics: perplexity training loss tokens per second wall-clock convergence peak GPU memory
Baselines: Decentralized uncompressed Centralized TopK low-rank SVD compression quantization
Builds on: AdamW GPipe Llama 3 TorchTitan Chinchilla scaling law
Closest papers by meaning
- Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training (NeurIPS 2025 · similarity 0.91)
- Practical Low-Rank Communication Compression in Decentralized Deep Learning (NeurIPS 2020 · similarity 0.90)
- Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.89)
- Towards a Better Theoretical Understanding of Independent Subnetwork Training (ICML 2024 · similarity 0.88)
- From Promise to Practice: Realizing High-performance Decentralized Training (ICLR 2025 · similarity 0.87)
- Forget the Data and Fine-Tuning! Just Fold the Network to Compress (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.