Research map / Image generation / Autoregressive and latent generation
PipeFusion: Patch-level Pipeline Parallelism for Diffusion Transformers Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PipeFusion, a patch-level pipeline parallelism for DiT inference that partitions layers and image patches across GPUs and reuses one-step-stale activations, cutting communication and memory versus TP, SP, and DistriFusion.
Problem
High latency of generating high-resolution images with diffusion transformers, where existing parallelism (tensor, sequence, DistriFusion) incurs large communication or memory costs.
Approach
Partition DiT layers and image patches jointly across GPUs in a patch-level pipeline, using stale activations from the previous diffusion step (input temporal redundancy) to provide context so devices never wait, so only the stage inputs/outputs are transferred instead of per-layer KV activations.
Key results
- PipeFusion is 1.48x, 1.55x, and 1.16x faster than the second-best parallel approach on PixArt at 1024px, 2048px and 4096px on 8xL40 PCIe GPUs.
- On FLUX.1-dev, PipeFusion achieves 1.23x and 1.25x speedups over the best USP configuration at 1024px and 2048px, and its memory is 32% and 36% of SP's.
- PipeFusion achieves lower FID than DistriFusion at matched device counts (e.g., 6.09 vs 19.81 w/ Orig. on PixArt at 8 devices).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PipeFusion instead of the second-best parallel method | Pixart inference on 8×L40 · Pixart 1024px, 2048px, and 4096px | speedup over second-best parallel method: 1.48×, 1.55×, and 1.16× | Helped |
| Use PipeFusion instead of the best sequence-parallel method | SD3-medium inference on 8×L40 · SD3-medium 1024px and 2048px | speedup over best sequence parallel method: 1.57× and 1.30× | Helped |
| Use PipeFusion instead of the best sequence-parallel method | Flux.1-dev inference on 8×L40 · Flux.1-dev 1024px and 2048px | speedup over best sequence parallel method: 1.23× and 1.25× | Helped |
| Replace PipeFusion with DistriFusion | Pixart image generation · PixArt-XL-2-256-MS, FID w/ G.T., device counts 4 and 8 | FID: 23.60 vs 30.69; 28.46 vs 41.81 | Hurt |
| Use 1 warmup step instead of 0 (ablation) | PipeFusion on Pixart · Pixart 1024px, pp=8 | latency: 0.71 -> 0.82 (+15%) | Hurt |
| Use 2 warmup steps instead of 0 (ablation) | PipeFusion on Pixart · Pixart 1024px, pp=8 | latency: 0.71 -> 0.91 (+28%) | Hurt |
| Set patch count M equal to GPU count N (ablation) | PipeFusion · Pixart image generation, multiple resolutions | latency | Helped |
| Increase warmup steps (ablation) | PipeFusion · Image generation | latency | Hurt |
Surprising findings
- Tensor parallelism and Ulysses sequence parallelism show high latency on PCIe, with Ulysses the slowest due to All2All bandwidth limits.
- On 8xA100 NVLink, the relative PipeFusion advantage narrows at higher resolutions as computation dominates.
Limitations
- Relies on multiple diffusion steps; minimal benefit for few-step distilled or one-step models such as Flux.1-schnell.
- Warmup steps create pipeline bubbles and cost latency, and the warmup setting is model-dependent.
- Evaluation limited to a single node of 8 GPUs; cross-node and video settings are not evaluated.
Experimental setup
Datasets: COCO Captions 2014
Benchmarks: COCO Captions 2014
Metrics: latency (s) FID max GPU memory speedup
Baselines: Tensor Parallel DistriFusion SP-Ring SP-Ulysses USP
Builds on: DistriFusion DeepSpeed-Ulysses Ring-Attention USP GPipe
Closest papers by meaning
- DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models (CVPR 2024 · similarity 0.92)
- Accelerating Parallel Diffusion Model Serving with Residual Compression (NeurIPS 2025 · similarity 0.90)
- Accelerating Diffusion Models with Parallel Sampling: Inference at Sub-Linear Time Complexity (NeurIPS 2024 · similarity 0.89)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.88)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.88)
- HarmoniCa: Harmonizing Training and Inference for Better Feature Caching in Diffusion Transformer Acceleration (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.