Research map / Architectures and efficiency / Attention and transformers
StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StarTrail, a concentric multi-ring sequence parallelism system that adds a team dimension to Ring Attention to cut peer-to-peer communication for near-infinite-context Transformer training.
Problem
Ring-style sequence parallelism for long-sequence Transformer training requires P-1 rounds of P2P communication carrying full keys and values, causing high communication overhead and bandwidth bottlenecks, while head-sharding methods are limited by the number of attention heads.
Approach
Group GPUs into teams that all-gather queries and key/values, then run ring communication over smaller subrings (P/C^2 iterations) with a reduce-scatter at the end, adding a tunable parallel dimension C that interpolates between Ring Attention (C=1) and collective-only attention (C=sqrt(P)).
Key results
- StarTrail reaches up to 2.114x the throughput of Ring Attention across six cluster settings on 32 GPUs.
- Reported speedups over Ring Attention reach up to 77.12% on GPT-style models and up to 114.33% on DiT models.
- For the highest-throughput configurations, StarTrail uses 7.9% to 30.79% more GPU memory than Ring Attention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Ring Attention with StarTrail sequence parallelism | Transformer training on the evaluated GPT-style and DiT-style configurations · GPT-style and DiT-style training; 32 GPUs across evaluated clusters | throughput: 1.199x to 2.114x | Helped |
| Use StarTrail-4 (C=4) instead of Ring Attention | Llama-30B · Llama-30B, N=65536, P=64 | communication volume: 1.625 GB -> 0.152GB (collective) + 0.406GB (P2P) = 0.558GB | Helped |
| Use the highest-throughput StarTrail configuration rather than Ring Attention | Transformer training on evaluated clusters · Evaluated throughput configurations across clusters | GPU memory use: 7.9% and 30.79% more GPU memory | Hurt |
| Vary the StarTrail team size C (Star-2 versus Star-4) (ablation) | StarTrail · Evaluated cluster and model configurations | throughput | Mixed |
Limitations
- Co-design with other hybrid parallelism strategies is not explored.
- Scaling is still limited by the inherent nature of ring-style communication, with reduced local sequence length at large GPU counts.
- Extra memory cost of QKV duplication grows relative to Ring Attention for smaller models.
Experimental setup
Metrics: throughput (tokens per second) peak GPU memory strong scaling weak scaling
Baselines: Ring Attention DeepSpeed-Ulysses Megatron Sequence Parallelism
Builds on: Ring Attention DeepSpeed-Ulysses FlashAttention Megatron sequence parallelism
Closest papers by meaning
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.86)
- Trainable Transformer in Transformer (ICML 2024 · similarity 0.86)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.86)
- A Multi-Level Framework for Accelerating Training Transformer Models (ICLR 2024 · similarity 0.86)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.86)
- RingAttention with Blockwise Transformers for Near-Infinite Context (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.