Research map / Optimization / Distributed and compressed

ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training

NeurIPS 2025 · Optimization › Distributed and compressed · approach: Gradient quantization and sparsification · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ACCO, a memory-efficient sharded AdamW optimizer that overlaps gradient communication with computation via a two-stage delayed-gradient compensation, matching DDP convergence while speeding up training and handling heterogeneous GPUs.

Problem

Data-parallel LLM training is limited by gradient communication overhead, while existing communication-overlapping methods (local-update, DPU) either break optimizer sharding, raise memory cost, or suffer convergence degradation from one-step delayed updates.

Approach

Split each worker's mini-batch into two halves: the first half estimates next-step parameters while communication runs, and the second half produces the actual update using both gradient sets, so that the parameter updates are computed synchronously with respect to the current parameters and avoid the staleness of DPU, while keeping optimizer states sharded.

Key results

• On GPT-Neo 125M pretraining with 4x8 A100 (50B tokens), ACCO reduced wall-clock time from 14h41m (ZeRO-1) to 10h55m, about 25% speedup.
• On GPT-Neo 2.7B instruction fine-tuning on Alpaca with 2x4 A100, ACCO time fell from 3h46m to 29min (reported as an 87% speedup).
• ACCO perplexity closely matches DDP on OpenWebText/LAMBADA (e.g., 1x8: 24.2 vs 24.3 on OpenWebText; 4x8: 22.5 vs 21.7).

Measured effects reported in the paper

Surprising findings

• DPU's one-step delay noticeably degrades LLM convergence even with warmup steps, and the warmup count used in prior work did not work well in the TinyStories experiments.
• Weight prediction (WP) shows worse training dynamics than DDP, especially at the start of training.

Limitations

• Local update methods and DPU are costly in memory; ACCO's advantage is mainly wall-clock and communication hiding, evaluated on small GPT-Neo models (125M, 2.7B) with limited scale.
• Heterogeneous hardware experiments are simulated with time.sleep() on a 4-worker setup rather than on real heterogeneous hardware.
• Theory is proven only for (S)GD on smooth functions, not for Adam/AdamW actually used in experiments.

Experimental setup

Datasets: TinyStories OpenWebText Pile Alpaca LAMBADA

Benchmarks: LAMBADA OpenWebText TinyStories

Metrics: perplexity training loss validation loss wall-clock time speedup

Baselines: DDP ZeRO-1 DPU WP (weight prediction) ZeRO-Offload

Builds on: ZeRO Delayed Parameter Update (DPU) Weight Prediction AdamW PyTorch DDP

Closest papers by meaning

• SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.88)
• Communication-Efficient LLM Adaptation over Decentralized GPU Meshes (NeurIPS 2026 · similarity 0.88)
• Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs (NeurIPS 2025 · similarity 0.87)
• Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training (NeurIPS 2026 · similarity 0.87)
• Revisiting Parameter Server in LLM Post-Training (ICLR 2026 · similarity 0.86)
• Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage (NeurIPS 2025 · similarity 0.86)