Research map / Architectures and efficiency / Attention and transformers
DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic Sequence Parallelism (DSP), which switches the sharded sequence dimension between computation stages of multi-dimensional transformers via all-to-all resharding, reducing communication versus embedded sequence parallel methods.
Problem
Long multi-dimensional sequences (e.g., video generation transformers with spatial-temporal attention) need sequence parallelism, but existing embedded methods shard along a single sequence dimension and incur large communication overhead.
Approach
Instead of sharding within a single sequence dimension and communicating inside modules, DSP dynamically switches the parallel dimension between computation stages using all-to-all resharding (Split/Switch/Gather primitives), decoupled from module logic.
Key results
- DSP improves end-to-end throughput over baselines from 32.2% to 10x on 128 H100 GPUs for 0.5M to 4M token sequences.
- DSP reduces communication volume by at least 50% (2M/N per layer vs 4M/N for DeepSpeed-Ulysses and 8M for Megatron-LM/Megatron-SP).
- DSP scales to 64 GPUs in weak scaling with only a 15% performance loss from 8 to 64 GPUs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DSP instead of DeepSpeed-Ulysses | 2D-Transformer · End-to-end performance, 0.5M to 4M tokens | throughput: 32% to 75% | Helped |
| Use DSP instead of other sequence parallel methods | 2D-Transformer · End-to-end performance, 0.5M to 4M tokens | throughput: up to 10x | Helped |
| Use DSP instead of DeepSpeed-Ulysses | 2D-Transformer · Per-layer communication volume per device | communication volume: $4M/N$ -> $2M/N$ | Helped |
| Use DSP instead of Ring Attention | 2D-Transformer · Per-layer communication volume per device | communication volume: $2M$ -> $2M/N$ | Helped |
| Use DSP instead of Megatron-LM or Megatron-SP | 2D-Transformer · Per-layer communication volume per device | communication volume: $8M$ -> $2M/N$ | Helped |
| Use DSP instead of other methods | 2D-Transformer · Weak scaling | throughput: more than 80.7% | Helped |
| Use DSP instead of other methods | 2D-Transformer · Scaling from 0.5M to 4M tokens | FLOPS drop: at most 23% vs at least a 40% | Helped |
| Use DSP instead of DS-Ulysses | 2D-Transformer · 8M sequence length | communication time (ms): 5828 -> 914 | Helped |
Surprising findings
- DSP shows super-linear scaling for intra-node weak scaling due to efficient communication.
Limitations
- Designed specifically for multi-dimensional transformers; may not suit single-dimensional models such as language models.
- Global operations involving all sequence dimensions may be less efficient under DSP.
Experimental setup
Metrics: throughput FLOPS communication volume inference latency memory consumption
Baselines: DeepSpeed-Ulysses Megatron-SP Megatron-LM Ring Attention
Builds on: DeepSpeed-Ulysses Megatron-SP Ring Attention ZeRO
Closest papers by meaning
- StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training (NeurIPS 2025 · similarity 0.84)
- MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (ICML 2025 · similarity 0.84)
- Approximation Rate of the Transformer Architecture for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.84)
- Transformers, parallel computation, and logarithmic depth (ICML 2024 · similarity 0.84)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.