Research map / Architectures and efficiency / RNNs and normalization

Communication-Efficient LLM Adaptation over Decentralized GPU Meshes

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a decentralized post-pretraining adaptation system that pairs 95% random in-graph activation masking on pipeline-parallel links with asynchronous unmasked "anchor" gradients and a spectral filter, plus DiLoCo/PowerSGD data-parallel compression.

Problem

Communication along data-parallel and pipeline-parallel axes is the bottleneck for training or adapting LLMs over low-bandwidth decentralized GPU meshes; naive activation compression degrades performance at high compression rates.

Approach

Use random PRF-based activation masking (no error feedback) on the fast circuit, and asynchronously denoise its gradients using delayed unmasked gradients from an anchor circuit, via an SVD-based soft spectral filter on an EMA of anchor gradients.

Key results

• M95+AP matches or exceeds dense baseline on most domain tasks (e.g., MedQA 0.321 vs 0.280 dense; HumanEval 0.207 vs 0.177) while M95 without anchor priors collapses (MedQA 0.180).
• M95 reaches roughly 45x TPS gain over the uncompressed baseline (48.9k vs 1.1k tok/s) by combining 20x PP masking with DP compression.
• On the SmolLM3-style three-phase recipe, M95+AP reaches IFEval 0.41 and GSM8K 0.21 vs dense 0.46 and 0.23 at ~43x throughput.

Surprising findings

• Random masking at 95% with anchor priors sometimes exceeds dense training on several tasks, which the authors attribute to a regularization effect.
• Two-bit activation compression baselines (AC-SGD, TAH-Quant) fail to learn at all on generative tasks such as HumanEval, MBPP, and GSM8K.

Limitations

• Experiments limited to fine-tuning regime; no claim for pretraining.
• Scale tested only 1B-8B parameters; 70B+ unexplored.
• Convergence analysis relies on assumptions (low-rank gradients, slow drift) that are argued rather than fully verified; staleness of 50 steps collapses performance.

Experimental setup

Datasets: MedQA MedMCQA HumanEval MBPP Spider ARC-C SciQ HellaSwag WinoGrande CNN/DM XSum GSM8K

Benchmarks: MedQA MedMCQA HumanEval MBPP Spider ARC-C SciQ HellaSwag WinoGrande CNN/DM XSum GSM8K

Metrics: accuracy pass@1 execution accuracy ROUGE-1 token F1 exact match tokens per second communication bytes per token

Baselines: AC-SGD TAH-Quant 4-bit quantization Top-k Bottleneck projection LoRA Dense SFT

Builds on: Streaming DiLoCo PowerSGD AQ-SGD GSPO LoRA

Closest papers by meaning

• Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism (NeurIPS 2025 · similarity 0.89)
• ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training (NeurIPS 2025 · similarity 0.88)
• ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload (NeurIPS 2026 · similarity 0.87)
• Practical Offloading for Fine-Tuning LLM on Commodity GPU via Learned Sparse Projectors (AAAI 2025 · similarity 0.87)
• Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs (NeurIPS 2025 · similarity 0.86)
• Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.86)