Research map / Architectures and efficiency / Attention and transformers
ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ZeCO, a sequence-parallelism method for linear attention (GLA) built on a pipelined All-Scan collective that passes only one state per device, achieving near-linear throughput scaling to 256 GPUs and 8M context.
Problem
Sequence parallelism for linear attention LLMs becomes a bottleneck because existing SP methods (LASP1 serial passing, LASP2 all-gather) incur communication overhead that grows with the number of devices, limiting scaling to ultra-long sequences.
Approach
A pipelined All-Scan collective that splits the inter-device recurrent state along the d_k dimension and forwards each block immediately after applying the decay-and-add update, so each device sends/receives only one state tensor and overlaps this with local intra-chunk diagonal attention; a cumulative-decay formulation lets local chunk states be corrected to global states without extra communicat
Key results
- All-Scan achieves up to 3.9x communication speedup; on 256 GPUs All-Gather (LASP2) communication is about 4x slower than All-Scan.
- On 256 GPUs with 16k sequence per GPU, ZeCO reaches 34400 tokens/sec/GPU throughput versus 15196 for LASP2 and 42838 for the GLA data-parallel baseline.
- The abstract claims ZeCO achieves 60% speedup over the SOTA SP method on 256 devices with 8M sequence length.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use All-Scan instead of All Gather Linear | Sequence-parallel communication · Communication Runtime, 256 GPUs | runtime: 8.51388 -> 2.16454 | Helped |
| Use ZeCO instead of LASP2 | GLA sequence-parallel operator · 16k sequence length, 128 GPUs | runtime: 35.72ms -> 9.88ms | Helped |
| Use ZeCO instead of LASP2 | GLA sequence-parallel operator · 32k sequence length, 128 GPUs | runtime: 42.50ms -> 15.06ms | Helped |
| Use ZeCO instead of LASP2 | 1B-GLA training · 1B-16k, 256 GPUs | throughput/GPU: 15196 -> 34400 | Helped |
| Use ZeCO instead of LASP2 | 1B-GLA training · 1B-32k, 256 GPUs | throughput/GPU: 25402 -> 40967 | Helped |
Surprising findings
- Overall LASP1 and LASP2 throughput degrades sharply with more GPUs, while ZeCO stays close to the data-parallel baseline.
Limitations
- Experiments only on a 1B GLA model with diagonal-decay gated linear attention; generalization to other linear attention forms is left as future work.
- The theoretical optimality analysis assumes worst-case overlap and treats extra compute/IO as negligible without full measurement.
- Pipelined All-Scan latency depends on the block count K; the paper gives no systematic tuning study.
Experimental setup
Metrics: communication runtime (ms) SP operator forward+backward runtime (ms) training throughput (tokens/sec/GPU)
Baselines: LASP1 LASP2 All-Gather (Megatron) Data parallelism (GLA DP upper bound)
Builds on: Gated Linear Attention LASP-1 LASP-2 Flash Linear Attention
Closest papers by meaning
- Sequence Accumulation and Beyond: Infinite Context Length on Single GPU and Large Clusters (AAAI 2025 · similarity 0.88)
- Various Lengths, Constant Speed: Efficient Language Modeling with Lightning Attention (ICML 2024 · similarity 0.88)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.87)
- ZeRO++: Extremely Efficient Collective Communication for Large Model Training (ICLR 2024 · similarity 0.86)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.