Research map / Architectures and efficiency / RNNs and normalization
TawPipe: Topology-Aware Weight Pipeline Parallelism for Accelerating Long-Context Large Models Training
Houming Wu, Ling Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add group-based weight pipeline scheduler (GWPS) (ablation) | TawPipe · C4, 24 GPUs, H=1024, S=16384 | throughput (kilo tokens/s): 8.59 (-3.6%) -> 8.91 | Improved |
| add communication-computation overlap (CCO) (ablation) | TawPipe · C4, 24 GPUs, H=1024, S=16384 | throughput (kilo tokens/s): 8.22 (-7.7%) -> 8.91 | Improved |
| add group-based weight pipeline scheduler (GWPS) (ablation) | TawPipe · C4, 24 GPUs, H=2048, S=16384 | throughput (kilo tokens/s): 3.91 (-6.5%) -> 4.18 | Improved |
| add communication-computation overlap (CCO) (ablation) | TawPipe · C4, 24 GPUs, H=2048, S=16384 | throughput (kilo tokens/s): 3.47 (-17.0%) -> 4.18 | Improved |
| add group-based weight pipeline scheduler (GWPS) (ablation) | TawPipe · C4, 24 GPUs, H=4096, S=16384 | throughput (kilo tokens/s): 1.26 (-8.7%) -> 1.38 | Improved |
| add communication-computation overlap (CCO) (ablation) | TawPipe · C4, 24 GPUs, H=4096, S=16384 | throughput (kilo tokens/s): 1.14 (-17.4%) -> 1.38 | Improved |
| use device-bound storage instead of ring-based weight buffers | TawPipe · weight buffer memory | weight buffer memory: 2M_W -> M_W | Improved |
| use TawPipe instead of WeiPipe as hierarchy-aware weight passing | TawPipe vs WeiPipe · C4, 24 GPUs, S=16384 | throughput: 6.0% to 11.8% as H scales from 1024 to 4096 | Improved |
| use TawPipe instead of WeiPipe for communication | TawPipe vs WeiPipe · C4, 24 GPUs, H=1024, S=16384 | NCCL execution time: up to 82.1% | Improved |
| use TawPipe instead of FSDP for communication | TawPipe vs FSDP · C4, 24 GPUs, H=1024, S=16384 | NCCL execution time: 16.8% | Improved |
| use TawPipe instead of 1F1B or FSDP at the most demanding configuration | TawPipe vs 1F1B/FSDP · C4, 24 GPUs, H=4096, S=16384 | throughput: 23.6%, 44.1% | Improved |
Closest papers by meaning
- CO2: Efficient Distributed Training with Full Communication-Computation Overlap (ICLR 2024 · similarity 0.89)
- Rethinking Memory and Communication Costs for Efficient Data Parallel Training of Large Language Models (NeurIPS 2024 · similarity 0.89)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.88)
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.88)
- Efficient Pre-Training of LLMs via Topology-Aware Communication Alignment on More Than 9600 GPUs (NeurIPS 2025 · similarity 0.87)
- SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
