Research map / Large language models / Agents, code and math
Synergistic Tensor and Pipeline Parallelism
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a synergistic tensor and pipeline parallel schedule that decouples forward/backward into fine-grained units, braids them into execution blocks to hide TP all-reduce communication, and uses a V-shape PP schedule to reduce pipeline bubbles, with an offloading variant for memory.
Problem
Hybrid TP+PP training of LLMs and MLLMs suffers from TP collective communication bubbles and PP pipeline bubbles, and existing methods address these in isolation.
Approach
Decouple forward and backward passes of each PP model chunk into fine-grained units (Pre-Attn, Attn, Pre-MLP, MLP, with backward split into activation and weight gradients) and braid forward units of one microbatch with backward units of another so TP all-reduce overlaps with computation; combine this with a V-shape virtual-stage dataflow for balanced memory and fewer PP bubbles.
Key results
- On 12.1B LLM across 16 A800 GPUs, the proposed schedule achieves up to 12.2% higher throughput than 1F1B-I (TP=8, seq 6144).
- On MLLMs, up to 16.7% throughput improvement over 1F1B-I in the PP=2 setting, and 11.7% at TP=8, PP=4.
- On 16 H20 GPUs, gains are smaller (e.g., 3.79 vs 3.77 samples/s for TP=2, PP=8) because TP communication is a smaller share.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the synergistic TP-PP schedule instead of 1F1B-I | Qwen2 12.1B · seq 6144, TP=8, PP=2, mbs=192 | throughput (samples/s): 3.11 -> 3.49 | Helped |
| Use the synergistic TP-PP schedule instead of ZB-V | Qwen2 26.3B · seq 4096, TP=8, PP=4, mbs=256 | throughput (samples/s): 4.29 -> 4.72 | Helped |
| Use the synergistic TP-PP schedule instead of 1F1B-I | Qwen2-VL 14.9B · ViT length 3136, LM length 5120, TP=8, PP=2, mbs=192 | throughput (samples/s): 2.46 -> 2.87 | Helped |
| Add activation offloading to the synergistic schedule (ablation) | Qwen2 12.1B · H20, TP=2, PP=8, mbs=1, seq_len=8192 | throughput: 2.72 -> 2.74 | Helped |
| Enable activation checkpointing on MLP (ablation) | Ours · Qwen2 12.1B, batch size 128, sequence length 6k | throughput (samples/s): 4.79 -> 4.19 | Hurt |
| Enable activation checkpointing on Attention and MLP (ablation) | Ours · Qwen2 12.1B, batch size 128, sequence length 6k | throughput (samples/s): 4.79 -> 3.94 | Hurt |
| Enable activation checkpointing on Attention, MLP, and Norm (ablation) | Ours · Qwen2 12.1B, batch size 128, sequence length 6k | throughput (samples/s): 4.79 -> 3.75 | Hurt |
| Begin with the memory-efficient warm-up phase (ablation) | Ours · Pipeline schedule | throughput | Hurt |
| Use activation offloading (ablation) | Ours · Qwen2 12.1B on H20 GPUs | throughput | Hurt |
Surprising findings
- ZB-V performs comparably to or worse than 1F1B-I because decoupling the backward pass exposes previously overlapped TP communication.
- The proposed schedule gives little benefit at PP=8 because its explicit pipeline communication cannot be overlapped with computation.
Limitations
- Peak activation memory is higher than baselines (e.g., 53 GB vs 41 GB for 1F1B-I), requiring an offloading variant under memory limits.
- Gains shrink on H20 GPUs with lower TP communication share and at PP=8 due to non-overlapped pipeline communication.
- Code released only after acceptance; no error bars reported.
Experimental setup
Metrics: throughput (samples/s) MFU peak activation memory (GB)
Baselines: 1F1B-I ZB-V
Builds on: Megatron-LM Zero Bubble 1F1B-I GPipe Flash Attention 2
Closest papers by meaning
- Tensor-Parallelism with Partially Synchronized Activations (NeurIPS 2025 · similarity 0.91)
- SPD: Sync-Point Drop for Efficient Tensor Parallelism of Large Language Models (ICML 2025 · similarity 0.89)
- FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training (NeurIPS 2025 · similarity 0.88)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.87)
- Heterogeneous Parallelism for Multimodal Large Language Model Training (NeurIPS 2026 · similarity 0.87)
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.