Research map / Architectures and efficiency / RNNs and normalization
Pipeline Parallelism with Controllable Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a building-block framework for pipeline schedules showing peak activation memory is set by building-block lifespan, and introduces V-shape schedules (V-Min, V-Half, V-ZB) with controllable memory that cut activation memory to about 1/3 or 1/2 of 1F1B.
Problem
Synchronous pipeline parallelism suffers from pipeline bubbles and large activation memory, and existing schedules lack a systematic methodology for controlling peak memory.
Approach
Decompose pipeline schedules as a repeating building block whose lifespan determines peak activation memory; use V-shape placement (2 stages per device, reversed second half) with split B/W backward passes and controlled offsets to balance and reduce memory while keeping bubbles low.
Key results
- V-Half reduces activation memory to about 1/2 of 1F1B (e.g., 19 GB vs 35 GB at 38.5B) with comparable throughput, and V-Min to about 1/3 (14 GB vs 35 GB at 38.5B).
- V-ZB achieves the highest throughput in pure pipeline settings (e.g., 61.3 vs 57.1 MFU for 9.6B at 256 microbatches), with bubble rate near 1-2% versus 8.8% for 1F1B.
- With grid search over hybrid parallelism on 40 GPUs, V-Half reaches 66.34% MFU versus 62.95% for 1F1B at sequence length 3072 (about 5% gain), and the abstract reports 16% overall improvement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use V-ZB instead of 1F1B | Pipeline schedule for 98.5B training · Sequence length 1024, batch size 160 | MFU (%): 54.77 -> 63.31 | Helped |
| Use V-Half instead of 1F1B | Pipeline schedule for 98.5B training · Sequence length 3072, batch size 640 | MFU (%): 62.95 -> 66.34 | Helped |
| Use V-Half instead of 1F1B-R | Pipeline schedule for 98.5B training · Sequence length 16384, batch size 160 | MFU (%): 42.05 -> 57.85 | Helped |
| Use V-Min instead of 1F1B | Pipeline schedule for 9.6B training · 16 GPUs, microbatch 4, 16 microbatches | Activation memory (GB): 46 -> 19 | Helped |
| Use V-Half instead of 1F1B-R | Pipeline schedule for 9.6B training · 16 GPUs, microbatch 4, 16 microbatches | MFU (%): 24.5 -> 38.3 | Helped |
| Use V-Min rather than V-Half | V-Shape pipeline schedules · Theoretical schedule, asymptotic | Peak activation memory: ≈ M/2 -> ≈ M/3 | Helped |
| Use V-Min rather than V-Half | V-Shape pipeline schedules · When F, B, and W have different run times and microbatch count grows | Bubble rate | Hurt |
| Use V-ZB rather than V-Half | V-Shape pipeline schedules · 98.5B, sequence length 3072, batch size 640 | MFU (%): 62.56 vs 66.34 | Hurt |
Surprising findings
- V-Min's throughput falls behind 1F1B at large microbatch counts due to a repeating bubble when F, B, W times differ.
- Full activation rematerialization (1F1B-R) is substantially slower than memory-saving V-Half and V-Min schedules.
Limitations
- V-Min suffers from repeating bubbles that grow with microbatch count when F/B/W run times differ.
- Uniform integer-unit offsets limit the space of building blocks explored.
- Evaluation limited to up to 40 A100 GPUs and GPT-3-like models.
Experimental setup
Metrics: MFU peak activation memory throughput (samples per second per GPU) bubble rate
Baselines: 1F1B interleaved 1F1B 1F1B-R ZB-1P ZB-2P
Builds on: 1F1B Zero Bubble pipeline parallelism Megatron-LM interleaved 1F1B GPipe
Closest papers by meaning
- PipeOffload: Improving Scalability of Pipeline Parallelism with Memory Optimization (ICML 2025 · similarity 0.90)
- Nesterov Method for Asynchronous Pipeline Parallel Optimization (ICML 2025 · similarity 0.84)
- AsyncMesh: Fully Asynchronous Optimization for Data and Pipeline Parallelism (NeurIPS 2026 · similarity 0.82)
- Synergistic Tensor and Pipeline Parallelism (NeurIPS 2025 · similarity 0.82)
- PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference (NeurIPS 2025 · similarity 0.81)
- Pfeife: Automatic Pipeline Parallelism for PyTorch (ICML 2025 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.