Research map / Architectures and efficiency / Attention and transformers
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Ahan Gupta, Zhihao Wang, Neel Dani, Masahiro Tanaka, Olatunji Ruwase, Minjia Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SP-aware AC pass on top of SP pass (ablation) | AutoSP · Max trainable tokens | max tokens before OOM: 77,000 -> 128,000 | Improved |
| add SP-aware AC pass on top of SP pass (ablation) | AutoSP · 40k sequence length | iteration time (s): 1.09 -> 1.19 | Hurt |
| replace hand-written DS-Ulysses with AutoSP SP pass only (ablation) | Llama-3.1 1B · Max trainable tokens | max tokens before OOM: 81,000 -> 77,000 | No clear effect |
| replace hand-written DS-Ulysses with AutoSP SP pass only (ablation) | Llama-3.1 1B · 40k sequence length | iteration time (s): 1.06 -> 1.09 | Hurt |
| replace DS-Ulysses with full AutoSP (SP + AC) | Llama-3.1 1B · Max trainable tokens | max tokens before OOM: 81,000 -> 128,000 (1.58x) | Improved |
| use AutoSP instead of ZeRO-3 (FSDP) with torch.compile | Llama 3B/8B/13B · Max sequence length on 8x A100 | max sequence length: upto 5x, 5.6x and 2.5x longer | Improved |
| use AutoSP instead of inductor-compiled DS-Ulysses | Llama 3B/8B/13B · Max sequence length on 8x A100 | max sequence length: upto 2.14x, 3x and 1.88x | Improved |
| use AutoSP instead of RingAttention | Llama 3B/8B/13B · Max sequence length on 8x A100 | max sequence length: upto 2.14x, 3x and 1.6x | Improved |
| use AutoSP instead of ZeRO-3 compiled baseline | Llama-3.1 8B · Per-iteration time, 8 GPUs | iteration time: 5x lower per-iteration time | Improved |
| use AutoSP instead of inductor baseline on NVIDIA and AMD | Llama-3.2 1B/3B · Runtime on GH200/MI250 | speedup: 0.97x, 0.98x (NVIDIA); 0.97x, 0.87x (AMD) | Mixed |
| use AutoSP instead of inductor baseline on diverse hardware | Llama-3.2 1B/3B · Max sequence length | max sequence length: 1.58x (2.70x) NVIDIA; 2x (2.5x) AMD | Improved |
| add AutoSP AC-pass rematerializing matmuls (ablation) | Llama-3.2 1B at 40k · Operator activation memory | memory reduction: 13.03x (attention), 2.22x (MLP) | Improved |
Closest papers by meaning
- Heterogeneous Parallelism for Multimodal Large Language Model Training (NeurIPS 2026 · similarity 0.90)
- TorchTitan: One-stop PyTorch native solution for production ready LLM pretraining (ICLR 2025 · similarity 0.90)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.89)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.89)
- The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements (NeurIPS 2025 · similarity 0.89)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
