Research map / Architectures and efficiency / RNNs and normalization

AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Distributed LLM inference systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Makes both data-parallel and pipeline-parallel axes asynchronous in a 2D mesh by pairing AsyncPP's Nesterov-style delay correction with 5% sparse, asynchronous weight averaging across replicas corrected by an EMA staleness estimate.

Problem

Data and pipeline parallel training require high-bandwidth co-located clusters because of synchronous communication; relaxing synchronization introduces weight staleness that must be corrected for replicas to reach consensus.

Approach

Asynchronous sparse averaging across replicas (only ~5% of weights, communicated asynchronously) with an EMA estimate of average staleness that reconstructs the new average from the delayed one, combined with AsyncPP delay correction within each pipe.

Key results

• On WikiText 4x2 mesh at 30k iterations, the method outperforms Async SPARTA by 3-6 perplexity points and matches FullSync, even surpassing it on 2 of 4 datasets.
• Trains a 1B parameter model in the AsyncMesh setup, reaching validation perplexity 19.53 vs 19.62 for FullSync on FineWeb.
• Asynchronous sparse averaging gives an estimated 1.5x-3.7x wall-clock speed-up over SPARTA on p4d instances, and tolerates up to 50-step delay with 5% subset.

Surprising findings

• Sparse averaging is robust to FP8 quantization while DPAvg diverges under FP8 and degrades even with BF16.
• The method diverges with subset sizes of 40% or more, yet performs similarly across 5%-30%.

Limitations

• Theory is proved only for a homogeneous setup with i.i.d. data and identical hyperparameters, with convergence in expectation.
• Learning rate theory suggests slow convergence for small subset sizes, which is not adjusted in practice.
• Experiments are simulated on 8 A100 instances, with asynchronous DP simulated via buffering rather than real decentralized deployment.

Experimental setup

Datasets: WikiText BookCorpus OpenWebText FineWeb

Benchmarks: WikiText BookCorpus OpenWebText FineWeb

Metrics: validation loss validation perplexity wall-clock speed-up

Baselines: FullSync DPAvg SPARTA Async SPARTA DiLoCo eager DiLoCo

Builds on: AsyncPP SPARTA DiLoCo GPipe NanoGPT

Closest papers by meaning

• FRESCO: A Novel Consistency Control for Asynchronous Pipeline Parallel Training (NeurIPS 2026 · similarity 0.92)
• Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation (ICML 2026 · similarity 0.90)
• Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.90)
• CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.87)
• Rescaled Asynchronous SGD: Optimal Distributed Optimization under Data and System Heterogeneity (NeurIPS 2026 · similarity 0.86)
• Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.86)